Method and apparatus for selectively integrating sensory content

ABSTRACT

To integrate a sensory property such as occlusion, shadowing, reflection, etc. among physical and notional (e.g. virtual/augment) visual or other sensory content, providing an appearance of similar occlusion, shadowing, etc. in both models. A reference position, a physical data model representing physical entities, and a notional data model are created or accessed. A first sensory property from either data model is selected. A second sensory property is determined corresponding with the first sensory property, and notional sensory content is generated from the notional data model with the second sensory property applied thereto. The notional sensory content is outputted to the reference position with a see-through display. Consequently, notional entities may appear occluded by physical entities, physical entities may appear to cast shadows from notional light sources, etc.

CLAIM OF PRIORITY

This application claims the benefit of U.S. Provisional Application No.62/074,771, entitled “METHOD AND APPARATUS FOR SELECTIVELY INTEGRATINGSENSORY CONTENT,” filed Nov. 4, 2014. This application is related toU.S. application Ser. No. 14/929,324, entitled “METHOD AND APPARATUS FORSELECTIVELY INTEGRATING SENSORY CONTENT,” filed Oct. 31, 2015. Theseapplications are incorporated herein in their entirety.

FIELD OF THE INVENTION

This invention relates to selectively integrating data models. Moreparticularly, the invention relates to selecting a sensory feature ofone data model and providing an analog of that sensory feature toanother data model.

DESCRIPTION OF RELATED ART

A data model may exhibit sensory properties that apply mainly or onlywithin that model, and that do not naturally propagate to other datamodels. For example, a data model reflecting a physical environmentmight include shadows cast from physical objects by a physical lightsource. However, another data model representing a notional environment(e.g. a virtual reality environment, an augmented reality environment,etc.) may not necessarily exhibit shadowing from the physical lightsources in the physical data model, even if notional data modelsimilarly includes shadows cast from notional objects by a notionallight source.

It may in certain instances be desirable to integrate sensory propertiesfrom one data model to another. For example, notional objects might bemade to case shadows corresponding to the illumination of physical lightsources, or vice versa, as if the notional and physical objects andlight sources were all part of a single environment. Such an arrangementmay for example facilitate interaction with both data modelssimultaneously and/or in cooperation, without necessarily requiring thedata models to be fully combined.

BRIEF SUMMARY OF THE INVENTION

This disclosure contemplates a variety of systems, apparatus, methods,and paradigms for facilitating data model interaction.

In one embodiment, a machine-implemented method is provided thatincludes, in a processor, establishing a reference position,establishing a physical data model at least substantially representingat least one physical entity, the physical data model being spatiallydynamic in time, and establishing a notional data model at leastsubstantially representing at least one notional entity, the notionaldata model being dynamic in time. The method includes determining anocclusion of the notional data model by the physical data model relativeto the reference position, wherein a first distance along a firstdirection from the reference position to the physical data model is lessthan a second distance along a second direction from the referenceposition to the notional data model, the second direction being at leastsubstantially similar to the first direction. The method also includesgenerating notional visual content representing at least a portion ofthe notional data model with at least a portion of the occlusion appliedthereto, and outputting the notional visual content to the referenceposition with a see-through display.

The first distance along the second direction from the referenceposition to the physical data model may be substantially equal to thesecond distance along the second direction from the reference positionto the notional data model.

The physical entity may include at least a portion of a viewer. Thephysical entity includes at least a portion of a hand of a viewer.

The notional entity may include a virtual reality entity. The notionalentity may include an augmented reality entity.

The reference point may substantially correspond with an eye of aviewer.

The see-through display may be a wearable display. The see-throughdisplay may be a head mounted display. The see-through display may bemay be an optical see-through display, such that optical physicalcontent at least substantially passes therethrough. The see-throughdisplay may be a virtual see-through display, with the method includingoutputting to the reference position with the see-through displayphysical visual content.

The notional data model may be non-exclusive of spatial coincidence withthe physical data model. The method may include outputting the notionalvisual content so as to be at least substantially registered withphysical visual content.

In another embodiment, a machine-implemented method is provided thatincludes, in a processor, establishing a reference position,establishing a physical data model at least substantially representingat least one physical entity, the physical data model being spatiallydynamic in time, and establishing a notional data model at leastsubstantially representing at least one notional entity, the notionaldata model being spatially dynamic in time. The method includesestablishing a first sensory property from one of the physical andnotional data models, and determining a second sensory property, thesecond sensory property at least substantially corresponding with thefirst sensory property for another of the physical and notional datamodels. The method also includes generating notional sensory contentrepresenting at least a portion of the notional data model with at leasta portion of the second sensory property applied thereto, and outputtingto the reference position with a perceive-through display the notionalsensory content.

The physical entity may be at least partially in line of sight from thereference position.

The first sensory property may include a visual property. The firstsensory property may include an auditory property. The second sensoryproperty may include a visual property. The second sensory property mayinclude an auditory property.

The physical entity may include at least a portion of a viewer. Thephysical entity may include at least a portion of a hand of a viewer.

The notional entity may include a virtual reality entity. The notionalentity may include an augmented reality entity.

The reference point may substantially correspond with an eye of aviewer.

The see-through display may be a wearable display. The see-throughdisplay may be a head mounted display. The see-through display may be anoptical see-through display, such that optical physical content at leastsubstantially passes therethrough. The see-through display may be avirtual see-through display, with the method including outputtingphysical visual content to the reference position with the see-throughdisplay.

The notional data model may be non-exclusive of spatial coincidence withthe physical data model. The method may include outputting the notionalvisual content so as to be at least substantially registered with thephysical visual content.

The method may include establishing the first sensory property from thephysical data model, with the second sensory property at leastsubstantially corresponding with the first sensory property for thenotional data model.

The first sensory property may at least substantially represent aphysical environmental phenomenon. The physical environmental phenomenonmay include physical illumination. The physical environmental phenomenonmay include physical shadowing. The physical environmental phenomenonmay include a physical volumetric effect. The physical volumetric effectmay be a function of the first distance between the reference point andthe physical data model. The physical volumetric effect may include afalling element, a flying element, and or a suspended element. Thephysical volumetric effect may include ash, debris, dust, fog, gas,hail, heat distortion, insects, leaves, mist, rain, sleet, smoke, snow,spray, and/or steam. The physical environmental phenomenon may includean optical phenomenon. The physical environmental phenomenon may includediffraction, diffusion, focus, glory, haloing, lens flare, and/orreflection.

The second sensory property effect may at least substantially representsa physical environmental phenomenon. The physical environmentalphenomenon may include physical illumination. The physical environmentalphenomenon includes physical shadowing. The physical environmentalphenomenon may include a physical volumetric effect. The physicalvolumetric effect may be a function of the first distance between thereference point and the physical data model. The physical volumetriceffect may include at least one of a group consisting of at least onefalling element, at least one flying element, and at least one suspendedelement. The physical volumetric effect may include at least one of agroup consisting of ash, debris, dust, fog, gas, hail, heat distortion,insects, leaves, mist, rain, sleet, smoke, snow, spray, and steam. Thephysical environmental phenomenon may include an optical phenomenon. Thephysical environmental phenomenon may include at least one of a groupconsisting of diffraction, diffusion, glory, haloing, lens flare, andreflection.

The method may include establishing the first sensory property from thenotional data model, with the second sensory property at leastsubstantially corresponding with the first sensory property for thephysical data model.

The second sensory property effect may at least substantially representsa physical environmental phenomenon. The physical environmentalphenomenon may include physical illumination. The physical environmentalphenomenon includes physical shadowing. The physical environmentalphenomenon may include a physical volumetric effect. The physicalvolumetric effect may be a function of the first distance between thereference point and the physical data model. The physical volumetriceffect may include at least one of a group consisting of at least onefalling element, at least one flying element, and at least one suspendedelement. The physical volumetric effect may include at least one of agroup consisting of ash, debris, dust, fog, gas, hail, heat distortion,insects, leaves, mist, rain, sleet, smoke, snow, spray, and steam. Thephysical environmental phenomenon may include an optical phenomenon. Thephysical environmental phenomenon may include at least one of a groupconsisting of diffraction, diffusion, glory, haloing, lens flare, andreflection.

The second sensory property may at least substantially represent thephysical environmental phenomenon.

The second sensory property effect may at least substantially representsa physical environmental phenomenon. The physical environmentalphenomenon may include physical illumination. The physical environmentalphenomenon includes physical shadowing. The physical environmentalphenomenon may include a physical volumetric effect. The physicalvolumetric effect may be a function of the first distance between thereference point and the physical data model. The physical volumetriceffect may include at least one of a group consisting of at least onefalling element, at least one flying element, and at least one suspendedelement. The physical volumetric effect may include at least one of agroup consisting of ash, debris, dust, fog, gas, hail, heat distortion,insects, leaves, mist, rain, sleet, smoke, snow, spray, and steam. Thephysical environmental phenomenon may include an optical phenomenon. Thephysical environmental phenomenon may include at least one of a groupconsisting of diffraction, diffusion, glory, haloing, lens flare, andreflection.

In another embodiment, an apparatus is provided that includes aprocessor and a perceive-through display in communication with theprocessor. A reference position establisher including executableinstructions is instantiated on the processor, the reference positionestablisher being adapted to establish a reference position. A physicaldata model establisher including executable instructions is instantiatedon the processor, the physical data model establisher being adapted toestablish a physical data model at least substantially representing atleast one physical entity, the physical data model being spatiallydynamic in time. A notional data model establisher including executableinstructions is instantiated on the processor, the notional data modelestablisher being adapted to establish a notional data model at leastsubstantially representing at least one notional entity, the notionaldata model being dynamic in time. An occlusion determiner includingexecutable instructions is instantiated on the processor, the occlusiondeterminer being adapted to determine an occlusion of the notional datamodel by the physical data model relative to the reference position,wherein a first distance along a first direction from the referenceposition to the physical data model is less than a second distance alonga second direction from the reference position to the notional datamodel, the second direction being at least substantially similar to thefirst direction. An occlusion applier including executable instructionsis instantiated on the processor, the occlusion applier being adapted toapply the occlusion to the notional data model. An outputter includingexecutable instructions is instantiated on the processor, the outputterbeing adapted to output to the reference position with theperceive-through display at least a portion of the notional data modelwith the occlusion applied thereto.

In another embodiment, an apparatus is provided that includes aprocessor and a see-through display in communication with the processor.A reference position establisher including executable instructions isinstantiated on the processor, the reference position establisher beingadapted to establish a reference position. A physical data modelestablisher including executable instructions is instantiated on theprocessor, the physical data model establisher being adapted toestablish a physical data model at least substantially representing atleast one physical entity, the physical data model being spatiallydynamic in time. A notional data model establisher including executableinstructions is instantiated on the processor, the notional data modelestablisher being adapted to establish a notional data model at leastsubstantially representing at least one notional entity, the notionaldata model being dynamic in time. A first sensory property establisherincluding executable instructions is instantiated on the processor, thefirst sensory property establisher being adapted to establish a firstsensory property from one of the physical and notional data models. Asecond sensory property determiner including executable instructions isinstantiated on the processor, the second sensory property determinerbeing adapted to determine a second sensory property at leastsubstantially corresponding with the first sensory property for anotherof the physical and notional data models. A second sensory propertyapplier including executable instructions is instantiated on theprocessor, the second sensory property applier being adapted to applythe second sensory property to the notional data model so as to generatenotional sensory content therefrom. An outputter including executableinstructions is instantiated on the processor, the outputter beingadapted to output to the reference position the notional sensory contentwith a perceive-through display.

In another embodiment, an apparatus is provided that includes means forestablishing a reference position, means for establishing a physicaldata model at least substantially representing at least one physicalentity, the physical data model being spatially dynamic in time, andmeans for establishing a notional data model at least substantiallyrepresenting at least one notional entity, the notional data model beingdynamic in time. The apparatus includes means for determining anocclusion of the notional data model by the physical data model relativeto the reference position, wherein a first distance along a firstdirection from the reference position to the physical data model is lessthan a second distance along a second direction from the referenceposition to the notional data model, the second direction being at leastsubstantially similar to the first direction. The apparatus alsoincludes means for generating notional visual content representing atleast a portion of the notional data model with at least a portion ofthe occlusion applied thereto, and means for outputting to the referenceposition with a see-through display the notional visual content.

In another embodiment, an apparatus is provided that includes means forestablishing a reference position, means for establishing a physicaldata model at least substantially representing at least one physicalentity, the physical data model being spatially dynamic in time, andmeans for establishing a notional data model at least substantiallyrepresenting at least one notional entity, the notional data model beingspatially dynamic in time. The apparatus includes means for establishinga first sensory property from one of the physical and notional datamodels, and means for determining a second sensory property, the secondsensory property at least substantially corresponding with the firstsensory property for another of the physical and notional data models.The apparatus also includes means for generating notional sensorycontent representing at least a portion of the notional data model withat least a portion of the second sensory property applied thereto, andmeans for outputting to the reference position with a perceive-throughdisplay the notional sensory content.

In another embodiment, a method is provided that includes establishing areference position in a processor, establishing in the processor solidsdata representative of an environment, and establishing in the processorsurfaces data representative of the environment. The method includesestablishing a notional data model in the processor. The method alsoincludes identifying in the processor at least one potential shadow inthe surfaces data, identifying in the processor at least one lightsource in the surfaces data, locating the light sources from thesurfaces data onto the solids data in the processor, and determining atleast one anticipated shadow in the processor from the solids data andthe light sources. If the potential shadow matches at least one of theanticipated shadows, the method includes identifying the potentialshadow as an occluding shadow in the processor, applying the occludingshadow to the notional model in the processor, generating in theprocessor notional sensory content representing at least a portion ofthe notional data model with the occluding shadow applied thereto, andoutputting the notional sensory content to the reference position with asee-through display.

Establishing the solids data may include obtaining with a depth sensorat least one depth measurement of the environment, communicating thedepth measurement to the processor, and determining the solids data fromthe depth measurement in the processor. The depth sensor may include adepth camera, a stereo camera, a plenoptic camera, a time-of-flightsensor, an ultrasonic sensor, and/or a millimeter wave sensor.

Establishing the surfaces data may include obtaining with an imagesensor at least one image of the environment, communicating the image tothe processor, and determining the surfaces data from the image in theprocessor. The image sensor may include a color camera, a grayscalecamera, a black-and-white camera, a digital camera, and/or an infraredcamera.

The light source may be explicit in the surfaces data and/or may beimplicit in the surfaces data.

A portion of the solids data casting anticipated shadow may be explicitin the solids data and/or implicit in the solids data.

A portion of the surfaces data may be persistent. A portion of thesolids data may be persistent.

The method may include identifying potential shadows from a colorfeature in the surfaces data and/or a structure feature in the surfacesdata. The method may include identifying potential shadows from analgorithm applied to the surfaces data.

The method may include identifying light sources from a color feature inthe surfaces data and/or a structure feature in the surfaces data. Themethod may include identifying light sources from an algorithm appliedto the surfaces data.

The method may include identifying potential shadows as not beingoccluding shadows from the solids data indicating solid structure to thepotential shadow.

The method may include determining if the potential shadow matches ananticipated shadow, determining a shadow correction in the processor,applying the shadow correction to the notional model in the processor,and generating in the processor notional sensory content representing atleast a portion of the notional data model with the shadow correctionapplied thereto.

In another embodiment, a method is provided that includes establishing aviewing position for a user in a digital processor, obtaining a digitaldepth image of an environment with a digital time-of-flight depth camerain communication with the processor, communicating the depth image tothe processor, determining in the processor solids data representativeof the environment from the depth image, obtaining a digital color imageof the environment with a digital color camera in communication with theprocessor, communicating the color image to the processor, anddetermining in the processor surfaces data representative of theenvironment from the color image. The method includes generating anaugmented reality data model in the processor. The method also includesidentifying in the processor at least one potential shadow from a colordistribution in the surfaces data, identifying in the processor at leastone light source from the color distribution in the surfaces data,locating the light sources in the surfaces data onto the solids data inthe processor, and determining at least one anticipated shadow in theprocessor from the solids data and the light sources. If the potentialshadow matches at least one of the anticipated shadows, the methodincludes identifying the potential shadow as an occluding shadow anddetermining a shadow correction for the potential shadow in theprocessor, applying the occluding shadow and the shadow correction tothe notional model in the processor, generating in the processoraugmented reality graphical content representing at least a portion ofthe augmented reality data model with the occluding shadow and theshadow correction applied thereto, and outputting the notional graphicalcontent to the viewing position with an optical see-through display incommunication with the processor.

In another embodiment, an apparatus is provided that includes aprocessor, and a see-through display in communication with theprocessor. The apparatus includes a reference position establisherincluding executable instructions instantiated on the processor, thereference position establisher being adapted to establish a referenceposition. The apparatus includes a solids data establisher includingexecutable instructions instantiated on the processor, the solids dataestablisher being adapted to establish solids data representing anenvironment. The apparatus includes a surfaces data establisherincluding executable instructions instantiated on the processor, thesurfaces data establisher being adapted to establish surfaces datarepresenting the environment. The apparatus includes a notional datamodel establisher including executable instructions instantiated on theprocessor, the notional data model establisher being adapted toestablish a notional data model. a potential shadow identifier includingexecutable instructions instantiated on the processor, the potentialshadow identifier being adapted to identify at least one potentialshadow in the surfaces data. The apparatus includes a light sourceidentifier including executable instructions instantiated on theprocessor, the light source identifier being adapted to identify atleast one light source in the surfaces data. The apparatus includes alight source locator including executable instructions instantiated onthe processor, the light source locator being adapted to locate thelight sources from the surfaces data onto the solids data. The apparatusincludes an anticipated shadow determiner including executableinstructions instantiated on the processor, the anticipated shadowdeterminer being adapted to determine at least one anticipated shadowfrom the light sources and the solids data. The apparatus includes ashadow matcher including executable instructions instantiated on theprocessor, the shadow matcher being adapted to match the potentialshadows against the anticipated shadows, and to identify the potentialshadows as occluding shadows if the potential shadows match at least oneof the anticipated shadows. The apparatus includes an occluding shadowapplier including executable instructions instantiated on the processor,the occluding shadow applier being adapted to apply the occludingshadows to the notional model. The apparatus includes a notional contentgenerator including executable instructions instantiated on theprocessor, the notional content generator being adapted to generatenotional content representing at least a portion of the notional datamodel with the occluding shadow applied thereto. The apparatus includesan outputter including executable instructions instantiated on theprocessor, the outputter being adapted to output notional content to thesee-through display.

The apparatus may include a depth sensor in communication with theprocessor, the depth sensor being adapted to obtain at least one depthmeasurement of the environment, wherein the solids data establisher isadapted to determine the solids data from the depth measurement. Thedepth sensor may include a depth camera, a stereo camera, a plenopticcamera, a time-of-flight sensor, an ultrasonic sensor, and/or amillimeter wave sensor.

The apparatus may include an image sensor in communication with theprocessor, the image sensor being adapted to obtain at least one imageof the environment, wherein the surfaces data establisher is adapted todetermine the surfaces data from the image. The image sensor may includea color camera, a grayscale camera, a black-and-white camera, a digitalcamera, and/or an infrared camera.

The processor and/or the see-through display may be disposed on a headmounted display.

The apparatus may include a shadow correction applier includingexecutable instructions instantiated on the processor, and being adaptedto apply a shadow correction for the potential shadow to the notionalmodel, wherein the notional content generator is adapted to generatenotional content representing at least a portion of the notional datamodel with the occluding shadow and the shadow correction appliedthereto.

In another embodiment, an apparatus is provided that includes means forestablishing a reference position, means for establishing solids datarepresentative of an environment, and means for establishing surfacesdata representative of the environment. The apparatus includes means forestablishing a notional data model. The apparatus also includes meansfor identifying at least one potential shadow in the surfaces data,means for identifying at least one light source in the surfaces data,means for locating the light sources from the surfaces data onto thesolids data, and means for determining at least one anticipated shadowfrom the solids data and the light sources. The apparatus includes meansfor identifying the potential shadow as an occluding shadow if thepotential shadow matches at least one of the anticipated shadows, meansfor applying the occluding shadow to the notional model, means forgenerating notional sensory content representing at least a portion ofthe notional data model with the occluding shadow applied thereto, andmeans for outputting the notional sensory content to the referenceposition in a see-through form.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

Like reference numbers generally indicate corresponding elements in thefigures.

FIG. 1 shows an example arrangement of physical, displayed notional, andperceived notional entities with respect to a reference point, in atop-down view.

FIG. 2A shows an example arrangement of physical and notional entitieswithout occlusion with respect to a reference point, in a top-down view.

FIG. 2B shows an example arrangement of physical and notional entitieswithout occlusion as viewed from a reference point.

FIG. 3A shows an example arrangement of a notional entity occluding aphysical entity with respect to a reference point, in a top-down view.

FIG. 3B shows an example arrangement of a notional entity occluding aphysical entity as viewed from a reference point.

FIG. 4A shows an example arrangement of a notional entitynon-geometrically occluding a physical entity with respect to areference point, in a top-down view.

FIG. 4B shows an example arrangement of a notional entitynon-geometrically occluding a physical entity as viewed from a referencepoint.

FIG. 5A shows an example arrangement of a physical entity occluding anotional entity with respect to a reference point, in a top-down view.

FIG. 5B shows an example arrangement of a physical entity occluding anotional entity as viewed from a reference point.

FIG. 6 shows an example method for integrating occlusion from a physicaldata model to a notional data model, in flow-chart form.

FIG. 7 shows another example method for integrating occlusion from aphysical data model to a notional data model with an emphasis onocclusion determination, in flow-chart form.

FIG. 8 shows another example method for integrating occlusion from aphysical data model to a notional data model with an emphasis onestablishing the physical model, in flow-chart form.

FIG. 9 shows an example arrangement of physical and notional entitiesand shadows thereof from a notional light source.

FIG. 10 shows an example arrangement of physical and notional entitiesand shadows thereof from a notional light source.

FIG. 11 shows an example arrangement of physical and notional entitiesand shadows thereof from physical and notional light sources.

FIG. 12 shows an example arrangement of physical and notional entities,a physical light source, a physical-physical shadow and aphysical-notional shadow.

FIG. 13 shows an example arrangement of physical and notional entities,a notional light source, a notional-physical shadow and anotional-notional shadow.

FIG. 14 shows an example arrangement of physical and notional entities,physical and notional light sources, and physical-physical,physical-notional, notional-physical, and notional-notional shadows.

FIG. 15 shows an example method for integrating a general sensoryproperty among a physical data model and a notional data model, inflow-chart form.

FIG. 16A and FIG. 16B show an example method for integrating a shadowingamong a physical data model and a notional data model, in flow-chartform.

FIG. 17 shows an example apparatus for integrating occlusion, inschematic form.

FIG. 18 shows another example apparatus for integrating occlusion inschematic form, with additional elements.

FIG. 19 shows an example apparatus for integrating a general sensoryproperty, in schematic form.

FIG. 20 shows an example apparatus for integrating shadowing, inschematic form.

FIG. 21 shows an example apparatus for integrating a sensory property inperspective view, in the form of a head mounted display.

FIG. 22 shows a block diagram of a processing system that may implementintegration operations.

FIG. 23 shows an example arrangement of a feature of unknown naturedisposed on a physical surface, in perspective view.

FIG. 24 shows an example arrangement of an augmented reality tabletdisposed above a physical surface, in perspective view.

FIG. 25 shows an example arrangement of an augmented reality tabletdisposed above a physical surface having an unknown feature thereon, inperspective view.

FIG. 26 shows an example arrangement of an augmented reality tabletdisposed above a physical surface, with a shadow on the tablet, inperspective view.

FIG. 27 shows an example arrangement of solids data for a physicallight, physical hand, and physical surface, in perspective view.

FIG. 28 shows an example arrangement of surfaces data for a physicallight, physical hand, physical surface, and unknown feature disposed onthe surface, in perspective view.

FIG. 29A and FIG. 29B show example arrangements illustrating an implicitlight source.

FIG. 30A and FIG. 30B show example arrangements illustrating an physicalentity, in perspective view.

FIG. 31A and FIG. 31B show example arrangements of shadow applicationand shadow correction, in perspective view.

FIG. 32A and FIG. 32B show an example method for integrating occlusionand/or shadowing from a physical data model to a notional data model, inflow-chart form.

FIG. 33A and FIG. 33B show another example method for integratingocclusion and/or shadowing from a physical data model to a notional datamodel, in flow-chart form.

FIG. 34A and FIG. 34B show an example method for integrating occlusion,shadowing, and/or shadow correction from a physical data model to anotional data model, in flow-chart form.

FIG. 35 shows FIG. 19 shows an example apparatus for integratingocclusion, shadowing, and/or shadow correction from a physical datamodel to a notional data model, in schematic form.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 1 shows an example arrangement of entities therein. A referenceposition 102 is represented by an eye of a viewer. Although shown as aneye for illustrative purposes, the reference position 102 is not limitedonly to eyes, nor even to sensors generally; the reference position 102is a point of view, and while an eye, other sensory organ, electronicsensor, etc. may be disposed at the reference position 102 this is anexample only, and other arrangements may be equally suitable.

A see-through display 104 is disposed proximate to and facing thereference position 102. As will be described in more detail laterherein, for visual models and/or entities notional visual content may beoutputted to the reference position 102 via a see-through display 104,and/or physical visual content may be outputted to the referenceposition 102 through a see-through display 104. However, the use of asee-through display in FIG. 1 is an example only, and other arrangementsmay be equally suitable.

The term “see-through display” 104 should be understood broadly herein.In a simple sense, a see-through display 104 may be a display that isphysically transparent, such that notional content is actively displayedthereon (e.g. using phosphors, LEDs, etc.) while physical content (e.g.the physical environment) is visible therethrough via opticaltransmission. However, other arrangements may be equally suitable. Forexample, embodiments may also incorporate what might be referred to as a“virtual see-through display”, wherein the physical environment isimaged in some fashion (e.g. with a video sensor) and then activelydisplayed.

Although for purposes of example embodiments may be described hereinchiefly with regard to vision, embodiments are not limited only tovisual phenomena. Other embodiments may enable cooperation of auditorymodels, for example, in which case an output device (if present) mightnot be commonly referred to as a “display” at all but perhaps rather asan audio speaker, etc. An example of such might be headphones delivernotional audio content while also passing physical audio contenttherethrough. So long as both the notional and physical sensory contentare delivered to the reference position, the manner and/or systemsutilized to do so may vary considerably. However for clarity the terms“display” and “see-through display” are used for at least certainexamples herein.

Continuing in FIG. 1, a physical entity 108 is shown in a position so asto be visible through the see-through display 104 from the referenceposition 102. In such fashion, physical visual content (in this examplea view of the physical entity) is delivered to the reference position102.

A notional entity 106 also is shown displayed by the see-through display104 from the reference position 102, such that notional visual content(in this example images on the display 104) is delivered to thereference position 102. An apparent notional entity 110 is shown aswell. The apparent notional entity 110 represents the appearance of thenotional entity 106 displayed by the see-through display 104. That is,although the actual output of the see-through display 104 typically issmall in size and is within, on the surface of, or near to the surfaceof the display 104 (as represented in FIG. 1 by the notional entity 106)the see-through display 104 may produce the appearance of content beingof a different size, at a different distance, etc. (as represented bythe apparent notional entity 110). For simplicity, subsequent figuresand description herein typically depict notional entities in theirapparent positions, rather than as images displayed within the display104. However, it should be kept in mind that although a notional entitymay be given the appearance of being distant from the reference point102, in practice notional entities typically are confined to beingoutputted at positions on, in, and/or or near the display 104.

The term “notional entity” is used herein broadly, to refer collectivelyto any entity that lacks concrete physical existence but that maynevertheless be sensed and/or otherwise manipulated. For example, acomputer generated 2D image, 3D object, wireframe model, audio clip,etc. might be argued as not being “real” in the sense of being physicalobjects, but may nevertheless be seen, heard, felt as solid (e.g. withhaptic inputs), etc. The term “notional entity” is applied hereinbecause such entities typically (though not necessarily) may existprimarily or entirely as data, rather than physical matter or energy.Notional entities include, but are not limited to, virtual realityentities and augmented reality entities.

Now with reference to FIG. 2A, another arrangement of a physical entity208A and a notional entity 210A relative to a reference position 202A isshown therein. (As noted previously with regard to FIG. 1, the notionalentity as outputted by a display is not shown, in favor of the apparentnotional entity referred to herein as 210A. Further, again forsimplicity the display itself is not shown in at least certain figuresherein, though in practice the display typically will be present.)

In addition, FIG. 2A includes sight lines 212A and 214A indicating thevisible extents of the physical entity 208A relative to the referenceposition 202A, and sight lines 216A and 218A indicating the visibleextents of the notional entity 210A relative to the reference position202A. (Strictly speaking, if the reference position is considered to bethe point of view for an actual eye, the reference position may beinside the eye, e.g. at or near the surface of the retina thereof wherethe images are focused by the eye's lens. For simplicity, issuesregarding internal structure and operation of the eye are not addressed;the precise location of the reference position may vary within the scopeof embodiments depending on particulars thereof.)

As may be observed with regard to FIG. 2A, sight lines 212A and 214A forthe physical entity 208A are not interrupted by the notional entity210A, nor are sight lines 216A and 218A for the notional entity 210Ainterrupted by the physical entity 208A. Thus, the physical entity 208Adoes not occlude the notional entity 210A, nor does the notional entity210A occlude the physical entity 208A.

Turning now to FIG. 2B, therein an arrangement with a physical entity208B and a notional entity 210B is also shown. The arrangement in FIG.2B is at least somewhat similar to the arrangement in FIG. 2A, althoughwhere FIG. 2A shows a top-down arrangement FIG. 2B shows a perspectiveview as might be visible from a reference position (the referenceposition not being itself illustrated in FIG. 2B). As may be seenperhaps more directly in FIG. 2B, the arrangement therein exhibitsneither occlusion of the physical entity 208B by the notional entity210B nor occlusion of the notional entity 210B by the physical entity208B.

The arrangements shown in FIG. 2A and FIG. 2B correspond with what maybe exhibited when a notional entity (e.g. a virtual image) is outputtedto a see-through display through which a physical entity also isvisible, with no overlap in position. That is, in the arrangements shownthe notional and physical entities are visually separated by somedistance, and so no occlusion is apparent. This may be considered abaseline case; no occlusion considerations arise since there is nooverlap that would be expected to result in occlusion.

With reference to FIG. 3A, an arrangement of a physical entity 308A anda notional entity 310A relative to a reference position 302A is showntherein. FIG. 3A also includes sight lines 312A and 314A indicating thevisible extents of the physical entity 308A relative to the referenceposition 302A, and sight lines 316A and 318A indicating the visibleextents of the notional entity 310A relative to the reference position302A.

As may be observed in FIG. 3A, sight lines 316A and 318A for thenotional entity 310A are not interrupted by the physical entity 308A.However, although sight line 312A for the physical entity 308A is notinterrupted by the notional entity 310A, sight line 314A is interruptedby the notional entity 310A. Thus, as viewed from the reference position302A the portion of the physical entity 308A between sight lines 316Aand 318A would be occluded by the notional entity 310A.

Now with reference to FIG. 3B, therein an arrangement with a physicalentity 308B and a notional entity 310B is also shown. The arrangement inFIG. 3B is at least somewhat similar to the that of FIG. 3A althoughFIG. 3B shows a perspective view as might be visible from a referenceposition. As may be seen in FIG. 3B, the arrangement therein exhibitsocclusion of the physical entity 308B by the notional entity 310B. Morecolloquially, the notional entity 310B appears to be closer to the pointof view (e.g. reference position) than the physical entity 308B, and thenotional entity 310B also appears to overlap part of the physical entity308B so as to block part of the physical entity 308B from view.

The arrangements shown in FIG. 3A and FIG. 3B correspond with what maybe exhibited when a notional entity (e.g. a virtual image) is outputtedto a see-through display through which a physical entity also isvisible, with the notional entity overlapping the physical entity sothat the physical entity is partially occluded by the notional entity.Considering, for example, an arrangement wherein a virtual realityobject is displayed with a see-through display, and a physical object isvisible through such a display, typically the virtual reality object mayoverlap the physical object, occluding some or all of the physicalobject. This is similar to what may be seen in FIG. 3A and FIG. 3B.

Although FIG. 3A and FIG. 3B show an arrangement wherein the notionalentity is solid and opaque, and thus completely obscures that portion ofthe physical entity that the notional entity overlaps, this is anexample only. If the notional entity were transparent, were an open wireframe, etc., the physical entity might be partially visible through thenotional entity. However, even in such instance, whatever visibility thenotional entity exhibits may still appear overlaid upon the physicalentity, and thus the notional entity may still be considered to occludethe physical entity. The particular appearance may vary from what isshown in the example arrangements of FIG. 3A and FIG. 3B (e.g. someportion of the physical entity may be visible as a “ghost” through atransparent notional entity), but the state of occlusion as shown anddescribed nevertheless may still apply.

With reference to FIG. 4A, another arrangement of a physical entity 408Aand a notional entity 410A relative to a reference position 402A isshown therein. FIG. 4A again includes sight lines 412A and 414Aindicating the visible extents of the physical entity 408A relative tothe reference position 402A, and sight lines 416A and 418A indicatingthe visible extents of the notional entity 410A relative to thereference position 402A.

As may be observed in FIG. 4A, sight lines 416A and 418A for thenotional entity 410A are not interrupted by the physical entity 408A.However, sight line 416A passes through the physical entity 408A, eventhough the physical entity 408A is closer to the reference position 402Athan is the notional entity 410A. In addition, even though the notionalentity 410A is farther from the reference position 402A than is thephysical entity 408A, sight line 414A is interrupted by the notionalentity 410A; thus the portion of the physical entity 408A between sightlines 414A and 416A is occluded by the notional entity 410A, and onlythe portion of the physical entity 408A between sight lines 412A and416A is visible at the reference position 402A without obstruction.

This arrangement may be viewed as counter-intuitive given the geometryshown in FIG. 4A. Namely, a more distant feature (the notional entity410A) appears in FIG. 4A to occlude a less distant feature (the physicalentity 408A). However, as noted previously with regard to FIG. 1, itshould be kept in mind that the notional entity 410A may be displayed ona see-through display that is in fact closer to the reference position402A than is the physical entity 408A.

With regard to FIG. 4B, therein an arrangement with a physical entity408B and a notional entity 410B is shown. The arrangement in FIG. 4B isat least somewhat similar to the that of FIG. 4A although FIG. 4B showsa perspective view as might be visible from a reference position. As maybe seen in FIG. 4B, the arrangement therein exhibits occlusion of thephysical entity 408B by the notional entity 410B. It is emphasized thenotional entity 410B partially occludes the physical entity 408B eventhough the notional entity 410B appears to be more distant than thephysical entity 408B, as also noted previously with regard to FIG. 4A.

An arrangement such as that in FIG. 4A and FIG. 4B, thoughcounter-intuitive, is possible in at least certain circumstances. Asnoted, a see-through display outputting a notional entity such as avirtual image may be closer to a point of view (or other referenceposition) than a physical object viewed through such a display.

One example of such arrangement would be a head mounted display usingoptically transparent screens, wherein the screens are disposed as closeas a few millimeters from the eyes but physical objects many metersdistant may be observed through the screens. For such an arrangement,the actual display output for notional entity might be only a fewmillimeters from the viewer's eyes, even when apparent position of thenotional entity is many meters distant.

When the display output for a notional entity is closer than a physicalobject, and the two overlap, then typically (absent otherconsiderations) the closer notional entity may occlude the more distantphysical entity, regardless of the perceived distance of the notionalentity based on other cues.

This may result in the counter-intuitive arrangements shown in FIG. 4Aand FIG. 4B, wherein the appearance is given of a closer physical entitybeing occluded by a more distant notional entity. (The apparent paradoxis, as noted, due to the notional entity actually being closer despiteappearances.)

However, although possible and geometrically explainable, an arrangementsuch as that in FIG. 4A and FIG. 4B may be problematic. For example, anotional entity may appear by some visual cues (focal distance, stereodistance, etc.) to be at one distance, while simultaneously appearing byvisual occlusion to be at a different distance. One possible result ofsuch dissonance among visual cues may be interruption of “suspension ofdisbelief”: if the notional entity is portrayed with the expectationthat it be seen as “real”, but does not behave as a real object, then aviewer's ability and/or willingness to act as though the notional entitywere a real object may be impaired. Other issues that might manifest mayinclude, but are not limited to, viewer discomfort or difficulties suchas disorientation, dizziness, nausea, eyestrain, headache, etc. Giventhe complexity of visual processes and (in both the eye and the brain)precise prediction of difficulties is itself problematic; someindividuals may exhibit some such issues, others entirely differentissues, still others no difficulties at all. Regardless, in at leastcertain instances it may be desirable to avoid such dissonant cues.

Now with reference to FIG. 5A, another arrangement of a physical entity508A and a notional entity 510A relative to a reference position 502A isshown therein. FIG. 5A again includes sight lines 512A and 514Aindicating the visible extents of the physical entity 508A relative tothe reference position 502A, and sight lines 516A and 518A indicatingthe visible extents of the notional entity 510A relative to thereference position 502A.

As may be observed in FIG. 5A, sight lines 514A and 516A are at leastsubstantially similar. In addition, a portion of the notional entity510A in FIG. 5A has been removed as compared with the notional entity410A in previously described FIG. 4A. More particularly, considering inFIG. 4A the portion of the notional entity 410A that occluded thephysical entity 408A therein, the corresponding portion of notionalentity 510A does not appear in FIG. 5A.

Arrangements for carrying out such removal as may be understood fromFIG. 5A are described in more detail subsequently herein. However, inessence the degree to which the physical entity 508A would occlude thenotional entity 510A based on the apparent position of the notionalentity 510A relative to the reference location 502A and the position ofthe physical entity 508A relative to the reference location 502A isdetermined, and a geometrically suitable occlusion of the notionalentity 510A is then applied to the notional entity. More colloquially,the portion of the notional entity 510A that should be blocked from viewby the physical entity 508A is removed from display.

Now with reference to FIG. 5B, therein an arrangement with a physicalentity 508B and a notional entity 510B is shown. The arrangement in FIG.5B is at least somewhat similar to the that of FIG. 5A although FIG. 5Bshows a perspective view as might be visible from a reference position.As may be seen in FIG. 5B, the arrangement therein appears to exhibitocclusion of the notional entity 510B by the physical entity 508B.However, the occlusion may be understood (with reference to FIG. 5A) tobe not a physical occlusion, but an applied occlusion. That is, thenotional entity 510B may be physically closer to a viewer than thephysical entity 508B (e.g. if a display that is outputting the notionalentity 510B is physically closer than the physical entity 508B), but theportion of the notional entity 510B that should be behind the physicalentity 508B is not displayed.

As a result, even though the overall arrangements in FIG. 5A and FIG. 5Bare otherwise similar to those in FIG. 4A and FIG. 4B, by applyinggeometrically suitable occlusion to the notional entity 510A and 510B inFIG. 5A and FIG. 5B the dissonant visual cues in FIG. 4A and FIG. 4B arenot present in FIG. 5A and FIG. 5B. Again more colloquially, andparticularly with reference to FIG. 5B, the notional entity “looksright” from the reference position.

FIG. 1 through FIG. 5B show example physical visual content and notionalvisual content in combination. In particular, FIG. 5A and FIG. 5B showan arrangement wherein physical visual content and notional visualcontent are integrated together, such that at least certain propertiesthat are exhibited within each data model—such as occlusion of distantobjects by closer objects—are maintained among the data models. That is,where occlusion may in at least certain circumstances be exhibited by aphysical object interacting with another physical object, or by anotional object interacting with another notional object, thearrangement in FIG. 5A and FIG. 5B shows occlusion exhibited by aphysical object interacting with a notional object. The feature ofocclusion thus is shown to be maintained when physical visual contentand notional visual content are integrated together.

With reference now to FIG. 6, an example arrangement is presentedwherein physical and notional visual content may be integrated, againwith reference to occlusion as an example. However, as will be describedin greater detail subsequently herein, embodiments are not limited onlyto exhibiting occlusion among integrated sensory content, and otherarrangements may be equally suitable.

Typically, though not necessarily, most or all steps in FIG. 6 (andsimilarly in other method figures herein) may be partially or entirelycarried out by and/or in cooperation with one or more processors. Insuch instance, the processor or processors may carry out method stepsthrough the execution of executable instructions instantiated thereon. Avariety of processors may be suitable for such tasks, and embodimentsare not limited with respect thereto.

In the method shown in FIG. 6, a reference position is established 620.In considering occlusion, one object, phenomenon, etc. typically isdisposed on a similar line of sight as that leading to some otherobject, phenomenon, etc. The reference position serves as an indicationof the base for the line of sight. More colloquially, in order toconsider whether one object obscures another, the point of view alsomust be considered. Thus the reference position may in some sense beconsidered to be a point of view. It is emphasized however that thereference position is not limited only to visual sensing, nor is aviewer necessarily required.

With regard to establishing the reference position 620, the term“establishing” is used broadly herein. It is noted that to “establish”something may, depending on particulars, refer to either or both thecreation of something new (e.g. establishing a business, wherein a newbusiness is created) and the determination of a condition that alreadyexists (e.g. establishing the whereabouts of a person, wherein thelocation of a person who is already present at that location isdiscovered, received from another source, etc.). Similarly, establishinga reference position may encompass several potential approaches, such asselecting or computing the reference position (e.g. through the use ofexecutable instructions instantiated on a processor), obtaining thereference position from a data store such as a hard drive or solid statedrive, receiving the reference position via communication with someexternal device, system, etc., and so forth. Other arrangements also maybe equally suitable, and embodiments are not limited with regard to howthe reference position is established 620.

Continuing in FIG. 6, a physical data model is established 622. Thephysical data model is an assembly of data, typically though notnecessarily instantiated on a processor, that describes at least oneparameter of at least one physical entity. For the particular example ofocclusion as described previously with regard to FIG. 1 through FIG. 5B,a physical data model may include information describing the position,dimensions, orientation, etc. of one or more physical entities such assolid material objects. However, as will be discussed subsequentlyherein in more detail embodiments are not limited only to featuresrelated to occlusion when facilitating data model interaction, and otherinformation than position, dimension, and/or orientation may be equallysuitable.

The term “physical entity” is used broadly herein. For simplicity atcertain points physical entities may be considered herein as beingphysical objects and/or surfaces, such as a hand, a wall, etc. However,substantially any object, phenomenon, or feature may serve as a physicalentity. For example, light sources and/or light therefrom, shadows, fog,smoke, heat haze, optical effects such as lens flare within an opticalsystem, etc. may for at least certain embodiments be considered andtreated as physical entities.

The physical data model is not limited only to reflecting physicalentities. Rules or patterns that describe behavior of physical objectsand phenomena, which may not necessarily be considered to be entitiesunto themselves, may nevertheless be part of a physical data model. Forexample, a description of or rules for Newtonian mechanical behaviormight be included within a physical data model (e.g. to assist inaccurate modeling of moving physical objects), even if Newtonianmechanical behavior were not considered to be a physical entity untoitself.

As noted with regard to establishing the reference position 620, theterm “establish” is used broadly with regard to establishing thephysical data model 622. Establishing the physical data model 622 mayencompass several potential approaches. For example, one or more sensorsmay collect information, with that information then being utilized forconstruction of a data model. As a more concrete example, a depth cameraor a pair of cameras in a stereo configuration might collect dataregarding the presence, shape, size, position, orientation, etc. ofphysical objects and phenomena in some area. Such information may thenbe used to construct a physical data model, for example through carryingout image processing and/or other calculations within a processor.However, other arrangements, including but not limited to obtaining aphysical data model (or information for constructing such) from a datastore such as a hard drive or solid state drive, and receiving aphysical data model (or information for constructing such) viacommunication with some external device, system, etc. Other arrangementsalso may be equally suitable, and embodiments are not limited withregard to how the physical data model is established 622.

Also with regard to terminology, it is noted that the term “physical”with regard to the physical data model does not indicate that the modelitself is a physical object or necessarily has physical substance.Typically though not necessarily, the physical model exists only as dataand/or executable instructions; the term “physical” should not be takento imply or require that the physical data model has some physicalsolidity, but rather to indicate the subject matter of the model (i.e.being reflective of one or more physical entities), and to distinguishfrom the notional data model.

The physical data model may be dynamic in time. That is, the physicaldata model may be adapted to change as time progresses, for example inresponse to changes in the physical environment such as objects moving,the reference position changing with respect to the physical model, newphysical objects or features being added, physical objects or featuresbeing modified, etc. Change may be continuous, periodic, and/or orsporadic; for certain embodiments it may be useful to update thephysical model at regular intervals to produce an appearance ofcontinuous change, at least somewhat similarly to the manner in which aseries of still frames of film may produce the appearance of acontinuously moving picture, but this is an example only and otherarrangements may be equally suitable.

Proceeding in FIG. 6, a notional data model is established 624. Thenotional data model is an assembly of data, typically though notnecessarily instantiated on a processor, that describes at least oneparameter of at least one notional entity. For the particular example ofocclusion as described previously with regard to FIG. 1 through FIG. 5B,a notional data model may include information describing the position,dimensions, orientation, etc. of one or more notional entities such as3D virtual objects. However, embodiments are not limited only tofeatures related to occlusion when facilitating data model interaction,and other information than position, dimension, and/or orientation maybe equally suitable.

As noted previously with regard to FIG. 1, the term “notional entity” isused broadly herein, to refer collectively to any entity that lacksconcrete physical existence but that may nevertheless be sensed and/orotherwise manipulated. Notional entities thus include but are notlimited to computer generated 2D images, 3D objects, audio clips, andother virtual reality and/or augmented reality objects and phenomena.

As noted with regard to establishing the physical data model 622, theterm “establish” is used broadly with regard to establishing thenotional data model 624. Establishing the physical data model 624likewise may encompass several potential approaches. For example, anotional model and/or information for constructing such may be generatedinternally within a processor, obtained from a data store such as a harddrive or solid state drive, received via communication with someexternal device, system, etc. Other arrangements also may be equallysuitable, and embodiments are not limited with regard to how thenotional data model is established 624.

Like a physical data model, a notional data model may include contentother than notional data entities, including but not limited to rulesdescribing behavior thereof.

However, unlike a physical data model, a notional data model mayliterally be or include the notional entity or entities in question,rather than being a reflection of one or more notional entities. If thenotional entities in question and the notional data model are bothinformational constructs, as may be true for at least certainembodiments, then the notional entities themselves (possibly includingrules for behavior of such entities, etc.) may serve as the notionaldata model. However, this is not required, and other arrangements may beequally suitable.

As noted with regard to the physical data model, the notional data modelalso may be dynamic in time. The notional data model thus may be adaptedto change as time progresses, for example in response to changes in thenotional environment such as objects moving, the reference positionchanging with respect to the notional model, new notional objects orfeatures being added, notional objects or features being modified, etc.Change may be continuous, periodic, and/or or sporadic.

Thus, having completed steps 620, 622, and 624 a reference position, amodel depicting at least one physical entity, and a model of at leastone notional entity are established. Such an arrangement may in at leastsome embodiments resemble the examples presented with regard to FIG. 1through FIG. 5B.

Continuing in FIG. 6, an occlusion of the notional model by the physicalmodel is determined 626 relative to the reference position. That is, thenotional model, the physical model, and the reference position areconsidered together, to determine whether some portion of the physicalmodel obstructs or should obstruct sight lines between the referenceposition and the notional model. Occlusion of the notional model by thephysical model might also be considered in terms of distances, e.g. isthe distance along a direction from the reference position to thephysical data model less than the distance along that direction to thenotional data model?

As noted previously with regard to FIG. 4A and FIG. 4B, if a notionalentity is displayed as visual output while a physical entity also isvisible, the displayed notional entity may overlap onto (i.e. occlude)the visible physical entity even if the nominal distance to the notionalentity according to the notional data model is greater than the distanceto the physical entity according to the physical data model. This mayoccur, for example, if the notional entity is made visible on a displaythat is closer to the viewer than is the physical entity. In suchinstance, conflicting visual cues may be provided.

However, through comparison of the reference position, the data modelfor the physical entity, and the data model of the notional entity, adetermination may be made as to what portion of the notional entity (ifany) should be occluded by the physical entity. For example, geometricalanalysis of distances along lines of sight from the reference positionto the physical data model and the notional data model may indicatewhether some portion of the notion model is behind (i.e. more distantthan) some portion of the physical model, from the point of view of thereference position.

For purposes of discussion, the portion of the notional entity in thenotional data model that is behind (more distant than) the physicalentity in the physical data model is referred to herein in at least someplaces as “an occlusion” and/or “the occlusion”, i.e. as a noun.

In determining the occlusion 626, at least for the example in FIG. 6what is being determined is a “thing” unto itself. While the occlusionmay not have physical substance, the occlusion nevertheless may be adata entity unto itself. For example, for certain embodiments theocclusion might be a two-dimensional geometric object in the form of amask, which may be applied to the notional data model if and when thenotional model is displayed in some fashion. However, an occlusionhaving such form is an example only, and other arrangements may beequally suitable.

Continuing in FIG. 6, notional sensory content is generated 628 from thenotional data model with the occlusion applied thereto. To continue theexample above, if the occlusion is a data entity in the form of a twodimensional geometric mask, that mask corresponding to the portion ofthe notional data model that is occluded by the physical data modelrelative to the reference position, then a visual representation of thecontents of the notional data model would then be generated (e.g. foroutput with a display), but with the occlusion applied thereto such thatthe occluded portion of the notional data model is not a visible part ofsuch a visual representation.

More colloquially, with regard to steps 626 and 628, the portion of thenotional data model that is occluded by the physical data model (andthat thus should appear to be occluded by the physical entity) isdetermined, and that portion of the notional data model is excluded frombeing displayed.

Applying the occlusion to the notional data model for purposes ofgenerating 628 notional sensory content should not be taken asnecessarily indicating that the occlusion and the notional data modelare combined in any sense except as sufficient to generate visualcontent with suitable occlusion of the notional data model. Inparticular, the occluded portion of the notional data model is notnecessarily deleted from the notional model, even though the occludedportion may not be visible in the sensory content as delivered to aviewer.

As noted previously herein with regard to step 626, for at least certainembodiments it may be useful to consider occlusion as a “thing” untoitself, whether that thing is a mask, a map, a defined area of space,etc. Such an arrangement may facilitate efficient execution of a methodfor certain embodiments.

As a more concrete example, consider an arrangement with athree-dimensional augmented reality environment and occlusion thereofwith respect to a three dimensional physical environment, as outputtedto a viewer using an optically transparent head mounted display. In suchcase, physical visual content passes through the optically transparenthead mounted display. However, augmented reality content may be renderedframe by frame, producing two dimensional rendered augmented realityimages from the three dimensional augmented reality environment (withthose images then being outputted to the viewer). Typically a physicalmodel is generated, for example through the use of a depth cameradisposed on the head mounted display, to provide information regardingthe physical environment so as to facilitate occlusion of the augmentedreality environment.

It may be useful for certain embodiments to combine the physical andnotional data models into a single model, or to treat the two datamodels together in rendering image output. In such instance, each frameof augmented reality imagery may be rendered based on analysis of thefull three dimensional augmented reality environment and the full modelof the three dimensional physical reality environment.

Such an approach might be referred to as a “pixel by pixel” process.That is, renders are generated considering both models in their entirety(or at least in substantial part), so as to produce two dimensionalimages of a simultaneous intersection of two three dimensional (orhigher dimensional) data models.

However, for at least some embodiments treating occlusion as an objectunto itself (as noted previously herein with regard to step 626) may beadvantageous. Rather than necessarily executing a full render of twothree dimensional models as described above, or combining two models andrendering the resulting combination, in generating and applying anocclusion as a distinct (or at least semi-distinct) feature from themodels themselves, generating graphical output may be made simpler andmore efficient.

Rather than necessarily generating occlusion within a renderpixel-by-pixel, the image changes—in the example of occlusion, theportions of the augmented reality environment that would be occluded bythe physical environment—may be rendered separately. The portions of theaugmented reality environment that are to be excluded may be determinedby segmenting nearby physical objects, through geometric analysis, orthrough other approaches. Such an approach may, for the examplecurrently under consideration, generate an occlusion in the form of atwo-dimensional mask frame that may be overlaid onto an augmentedreality render frame, with areas under the mask not being outputted tothe head mounted display.

Regardless of such particulars however, with the occlusion generated,the occlusion may be applied to, combined with, etc. the augmentedreality visual content that is to be delivered to the viewer.

Thus, as noted previously, for at least certain embodiments it may beuseful to treat occlusion as a “thing” unto itself. However, this is anexample only, and other arrangements may be equally suitable.

In addition, with reference to step 628, in the example embodiment ofFIG. 6 the occlusion is applied in generating content from the notionaldata model. However, for at least certain embodiments it may be equallysuitable to apply the occlusion in generating content from the physicaldata model, and/or in generating content from both models. The occlusionalso may be used to generate occlusion content entirely separately, anddelivered as a third “stream” of content. Embodiments are not limited tothe manner in which the occlusion is made to appear as visual content.However, since at least certain embodiments may utilize an opticalsee-through display, wherein no physical visual content is generated(rather, light reflecting from physical objects in such case beingpassively transmitted through the optical see-through display), forpurposes of clarity the occlusion is described chiefly herein as beingapplied to content generated from the notional data model (i.e. becausecontent is not necessarily generated from the physical data model).

Continuing in FIG. 6, physical sensory content is outputted 630 to thereference position. For an embodiment addressed to an opticalsee-through display, outputting 630 the physical sensory content may bea passive step, i.e. what physical objects exist are allowed to passthrough the see-through display so as to be visible therethrough.However, as has been noted at least certain embodiments may addressvirtual see-through displays, wherein the physical environment is imagedwith for example a camera and outputted to a display that is physicallyopaque. For such embodiments, outputting 630 physical sensory contentmay for example include outputting such images of the physicalenvironment to the display as an active step (e.g. outputting images ofthe physical environment on an LED screen, OLED screen, using a scanninglaser, etc.).

Thus, for at least certain embodiments step 630 may be optional, atleast insofar as being an active step. (Light passing through anoptically transparent display may still in some sense constitute“output”, but may not require active execution as part of a method.)Step 630 is included in FIG. 6 for completeness, but not all embodimentsmay require such a step.

The notional sensory content is also outputted 632 to the referenceposition. As noted with regard to step 628, the occlusion is applied tothe notional sensory content therein. Thus, the notional sensory contentas outputted 632 to the reference position may not include a visualrepresentation of that portion of the notional data model that isoccluded by the physical data model.

Thus, if some portion of the physical data model occludes some portionof the notional data model, then as shown in FIG. 6 the notional sensorycontent is displayed with the appearance that the actual physicalenvironment occludes the relevant portion of the notional environment.(As noted previously, for portions of the notional data model that mayocclude the physical data model, the notional sensory content mayocclude the relevant portion of the physical environment naturally,without modification being required.)

In other words, where notional content has a nominal distance (based onnotional and physical data models) that is “behind” physical content,that notional content is not displayed. Thus an intuitively correctappearance is provided wherein, the notional content appears to bebehind the physical content. As previously noted, for certainembodiments such intuitively correct appearance may be enabled eventhough in terms of physical geometry the notion content is displayed infront of the physical environment (e.g. because the notional content isdisplayed on a head-mounted display that is disposed near a viewer'seyes).

Typically, the notional content as outputted is registered with thephysical content. That is, for example a virtual reality shadow outputso as be (or appear to be) cast by a physical object may be aligned withthe physical object in similar fashion to a real shadow. Registrationmay consider factors including but not limited to position (whether intwo dimensions, three dimensions, etc.), scale, shape, etc. Perfectregistration may not be necessary; for example, some degree of mismatchbetween the aforementioned virtual shadow and physical object may beacceptable, e.g. if registration is sufficient that the mismatch doesnot distract the viewer, or produce imaging artifacts, or cause otherissues, etc., though the precise degree of mismatch and the degree(s) towhich negative side effects may be acceptable may vary amongembodiments.

Although FIG. 6 shows the method therein as being complete followingstep 632, other steps, other functions, etc. may be incorporated intothe method, and/or other methods may be executed in combination with themethod (and this applies likewise to other example methods shown herein,and to the method generally). Notably, for at least certain embodimentsat least some portion of the method may repeat. For example, an ongoingloop may accommodate changes in the physical data model and/or notionaldata model which as stated previously may be dynamic over time. Thus thephysical and notional sensory content likewise may be dynamic in time.

Now with reference to FIG. 7, another example arrangement is presentedwhereby physical and notional visual content may be integrated, againwith reference to occlusion as an example. However, where the examplemethod in FIG. 6 showed a general method, the example method in FIG. 7provides an example of more particular approaches for certain stepswithin the method, and also addresses the method to use with a specificform of apparatus, namely an optically transparent head mounted displayadapted for augmented reality output. It is emphasized that embodimentsare not limited only to such an apparatus, and that other arrangementsmay be equally suitable.

In FIG. 7, a viewing position is established 720 for the head mounteddisplay. In the example of FIG. 7, the viewing position typicallycorresponds to an anticipated position of a viewer's eye or eyes, e.g.his or her retina(s), relative to the head mounted display. For purposesof simplicity the method shown in FIG. 7 addresses only a single suchviewing position. However, this is an example only, and otherarrangements may be equally suitable. For example, an arrangementwherein two distinct viewing positions are considered having a stereoarrangement, with occlusion determined for each one and correspondingchanges to visual content as delivered to each viewing position, may besuitable for at least certain embodiments. Such an arrangement maysupport the appearance of three dimensionality for visual contentdelivered to a viewer, and/or provide other advantages, but such astereo arrangement is not required. Alternately, multiple non-stereoviewing positions may be utilized, for example one for an infrared depthcamera and one for an RGB camera, or some other arrangement.

Typically though not necessarily, the viewing position is established720 at least in preliminary form in advance, based on the configurationof the head mounted display and the anticipated position of the headmounted display relative to the viewer. For example, given a headmounted display in the form of a pair of glasses and adapted to be wornin similar fashion, viewing positions suited for delivering visualcontent to a wearer may be at least approximated.

However, also typically though not necessarily, the viewing position maybe established 720 in part through adjustment and/or calibration of thehead mounted display. For example, the physical configuration of thehead mounted display may be adjusted based on the particulars of anindividual wearer's anatomy, preferences, etc. (e.g. extending arms,adjusting a nose support, etc.). Other adjustments, including but notlimited to optical adjustments such as altering the focus of thedisplay(s), likewise may be suitable.

Step 720 does not necessarily include such manipulations (though suchmanipulations also are not necessarily excluded). Rather, establishing720 the viewing position for the head mounted display addressesacquiring information as to where the desired viewing position is inspace, and providing such information to a processor (e.g. a processorcontrolling determination of occlusion, output of visual content, etc.for the head mounted display) if such information is not inherentlypresent (as may be the case if the processor itself calculates theviewing position using executable instructions instantiated thereon).

Continuing in FIG. 7, a physical visual model is established 722, thephysical visual model typically representing at least a portion of thephysical reality environment proximate the head mounted display. Thatis, the physical visual model represents the visual appearance of “thereal world” in the region around the head mounted display. Such a modelmay be constructed for example through evaluating stereographic images,depth maps, etc. as may be acquired through cameras, depth cameras,and/or other sensors (e.g. sensors on or in the head mounted display).However, other arrangements may be equally suitable.

The physical visual model may be dynamic over time.

An augment visual model is established 724, the augment visual modelrepresenting an augmented reality environment, the augmented realityenvironment typically though not necessarily being proximate the headmounted display. An augmented reality environment typically providesinformation, emphasis, etc. relating to physical reality. For example,an augmented reality environment might include direction arrows overlaidonto a sidewalk indicating a path to some destination, blinking boxesaround relevant street signs to draw attention thereto, a floatingindicator displaying the time until the next bus arrives at someposition, etc. An augmented reality typically is constructed within aprocessor, including but not limited to a processor in the head mounteddisplay, using executable instructions instantiated thereon and/or dataaccessible thereto. As has been noted previously, an augment visualmodel (or another notional model) may in fact be the augmented realitycontent itself; that is, where a processor typically could not contain aphysical reality environment, an augmented reality environment may becontained within a processor. Thus, establishing 724 the augment visualmodel may not imply creating such a model, but rather identifying asuitable model, accessing a model, etc.

The augment visual model also may be dynamic over time.

Still with reference to FIG. 7, the depth from the viewing position tothe physical visual model is established 726A along a direction.Typically though not necessarily, the depth to the physical visual modelmay be measured and/or calculated from the physical visual model and thereference position. For example, if the reference position is known, andthe geometry of the physical visual model with respect to the referenceposition is known, then the distance from the reference position to thephysical visual model along any direction (and thus to any point in thephysical visual model) may be determined through geometric analysis,e.g. as performed in a processor using executable instructionsinstantiated thereon. However, other arrangements may be equallysuitable. For example, if the physical visual model is established 722as a depth map (e.g. by utilizing a depth camera), then the depth tovarious points in the visual model may already be substantially orentirely known. (For certain embodiments a correction may be applied toaccount for the depth camera being disposed some distance and directionfrom the viewing position.) That is, the depth along one or moredirections may already be part of the physical visual model, or even thebasis for establishing 722 the physical visual model. Thus for at leastcertain embodiments the step of establishing 726A the depth to thephysical visual model along a direction may be implicit in step 722(and/or other steps), and may or may not be a completely distinct step.Nevertheless, so long as the depth to the physical model is in somemanner established 726A, embodiments are not limited with regard to howsuch depth is established.

Again typically but not necessarily, the depth to the physical visualmodel may be established 726A along a plurality of directions. Forexample, depth might be established 726A in particular along directionsat which physical content and augmented reality content begin or end aregion of overlap. Alternately, a grid or other arrangement of manydirections from the viewing position to the physical visual model may beused to determine depth thereto over areas or even the entirety of thephysical visual model. Other also arrangements may be equally suitable.

Continuing in FIG. 7, the depth from the viewing position to the augmentvisual model along the direction(s) is established 726B. Forarrangements wherein depth is so established 726B along only a singledirection, the direction in step 726B may be at last substantiallysimilar to the direction in step 726A.

However, in particular when many depths along many directions areestablished in steps 726A and/or 726B, a one-to-one correspondencebetween directions for steps 726A and 726B is not required (though suchcorrespondence also is not prohibited). Depths may be established alongsome directions in step 726A but not along substantially similar orotherwise corresponding directions in step 726B, or vice versa.

In similar fashion as noted with regard to step 726A, the depthsestablished in step 726B may be established through geometric analysisor other calculation with a processor (e.g. with executable instructionsinstantiated thereon), may be inherent in the augment visual model asestablished in step 724, or may be established in some other fashion.

Continuing in FIG. 7, a determination is made 726C as to whether thedepth to the physical visual model (which may be referred to as thephysical depth) is less than the depth to the augmented visual model(which may be referred to as the augment depth) along substantiallysimilar direction(s) as established in steps 726A and 726B. This may becarried out for example through scalar comparison of the depthsestablished in steps 726A and 726B, e.g. in a processor with executableinstructions instantiated thereon. However, other arrangements may beequally suitable.

If the determination in step 726C is positive—if the physical depth isless than the augment depth—then the method continues to step 726D(below). If the determination in step 726C is negative—if the physicaldepth is not less than the augment depth—then the method skips step 726Dand continues with step 728.

Continuing in FIG. 7, the augment visual model is indicated 728D asbeing occluded by the physical visual model along the direction inquestion. Typically, though not necessarily, multiple such directionswill be evaluated, thus producing a “map” of what portion(s) of theaugment visual model are to be indicated 728D as occluded. However,other arrangements, including but not limited to indicating 728D asoccluded entire regions and/or features of the augment visual model,entities within the augment visual model, or even the entirety of theaugment visual model, etc. on the basis of a single direction whereinthe determination in step 726C is that the physical depth is less thanthe augment depth.

Indication may vary considerably between embodiments. One examplearrangement for indication might be to apply a digital “tag” to pixels,entire entities, etc. indicating a state of occlusion, so that output ofsuch tagged pixels, entities, etc. may be excluded. Another examplearrangement might be to define a geometric region of the augment visualmodel and similarly apply a tag to that region, tagged regions likewisebeing excluded from output. However, these are examples only, and otherarrangements may be equally suitable.

Steps 726A, 726B, 726C, and 726D may be considered collectively as beingat least somewhat comparable to step 626 in FIG. 6 wherein occlusion ofa notional model by a physical model is determined.

Again with reference to FIG. 7, augment visual content is generated 728,using as source material the augment visual model and the occlusionindications. Thus, at least a portion of the augment visual model isconfigured as data suitable for visual output, with those portionsindicated for occlusion (e.g. tagged) as in step 726D being excludedfrom visual output. Typically, though not necessarily, the augmentvisual content may be two-dimensional so as to be adapted for outputusing a two-dimensional display (even though the augment visual modeland/or the indication(s) of occlusion may be three-dimensional or ofsome other form).

Also typically though not necessarily, the augment visual content may begenerated within a processor, such as one having suitable executableinstructions instantiated thereon. For example, a graphical engine mightrender a frame (or series of frames) showing the contents of the augmentvisual model in visible form as would be seen from the viewing positionand bounded within some field of view. However, other arrangements maybe equally suitable.

As previously noted, the example method of FIG. 7 is presented asspecific to use with an optically transparent head mounted display.Thus, for the particular example of FIG. 7 physical visual content maybe considered to pass through the display to the viewing positionwithout an active step in the example method. This passive opticaltransmission may be in some sense analogous to step 630 in FIG. 6,though as noted the method does not require action.

Continuing in FIG. 7, the augment visual content is outputted 732 to theviewing position with the head mounted display. For example, for a headmounted display utilizing one or more LED screens, the augment visualcontent may be displayed as an arrangement of pixels forming a visibleimage. To continue the example presented with regard to step 728, such avisible image may depict the contents of the augment visual model asseen from the viewing position and bounded within a field of view (e.g.a field of view substantially corresponding with the edges of the LEDscreen). However, this is an example only, and other arrangements may beequally suitable.

Although FIG. 7 shows the method therein as being complete followingstep 732, other steps, other functions, etc. may be incorporated intothe method, and/or other methods may be executed in combination with themethod. For example, as noted with regard to FIG. 6 for at least certainembodiments at least some portion of the method may repeat. Eachrepetition may yield a new still image displayed on the head mounteddisplay, so that the visual augment content as visible to a viewer maybe dynamic in time. A sufficiently rapid rate of update may for exampleproduce the appearance of continuous change and motion in the visualaugment content, presenting at least the impression of a smoothlyvariable, interactive, responsive visual experience for the viewer.

Now with reference to FIG. 8, another example arrangement is presentedwhereby physical and notional visual content may be integrated, againwith reference to occlusion as an example. The example method in FIG. 8provides an example at least somewhat similar to that of FIG. 6 butagain having more particular approaches for certain steps within themethod, and also addresses the method to use with a specific form ofapparatus, namely a virtually transparent head mounted display adaptedfor augmented reality output.

In FIG. 8, a viewing position is established 820 for the head mounteddisplay. As noted with regard to FIG. 7, for the example embodiment ofFIG. 8 the viewing position typically corresponds to an anticipatedposition of a viewer's eye or eyes, e.g. his or her retina(s), relativeto the head mounted display. Establishing a viewing position, andcertain features and options with regard thereto, were previouslydescribed with respect to FIG. 7.

Continuing in FIG. 8, at least one camera image is obtained 822A, thecamera image depicting at least a portion of a physical environment.Such images may be obtained for example from one or more camerasdisposed on the head mounted display, such that the field of view of thecameras at least substantially aligns with the field of view of theviewer. Typically the images will in some fashion include informationregarding depth to the physical environment. For example, two camerasdisposed in a stereo arrangement on the head mounted display may supportobtaining stereo pairs of images, which together may incorporate threedimensional data therein (even if the images themselves are twodimensional). Alternately, a depth camera may capture both an image anddepth data within the image. These are examples only, and otherarrangements also may be equally suitable.

The camera image is received 822B in a processor. Typically though notnecessarily this processor will be disposed on the head mounted display.However, other arrangements including but not limited to externalcameras or some combination of cameras on the head mounted display andelsewhere may be equally suitable. Embodiments are not limited withregard to how the images are received in the processor; wirelesscommunication, direct wired connection, or some other route may beequally suitable.

A physical visual model is generated 822C with the processor from thecamera images and/or data therein. The physical visual model typicallyrepresents at least a portion of the physical reality environmentproximate the head mounted display. That is, the physical visual modelrepresents the visual appearance of “the real world” in the regionaround the head mounted display. However, other arrangements, includingbut not limited to physical visual models that represent distantphysical environments, or physical visual models generated from imagesstored at a previous time, may be equally suitable.

The physical visual model may be generated 822C from the camera imagesfor example through stereographic analysis of stereo image pairs,through considering depth data from a depth camera image to construct athree dimensional model, etc. Other arrangements also may be equallysuitable.

The physical visual model may be generated 822C so as to be dynamic intime. For example, the model may be generated 822C such that if and whennew camera images are received, the physical visual model is modified orreplaced so as to reflect the new state of the physical environment asreflected by the new camera images.

Steps 822A, 822B, and 822C may be considered collectively as being atleast somewhat comparable to step 622 in FIG. 6 wherein a physical depthmodel is established.

Proceeding in FIG. 8, an augment visual model is established 824, theaugment visual model representing an augmented reality environment thatis typically though not necessarily proximate the head mounted display.The augment visual model may be dynamic over time. An augment visualmodel has been previously described herein with regard to FIG. 7.

A determination is made 826 of the visual occlusion (if any) of theaugment visual model by the physical visual model. That is, for examplebased on the geometry, arrangement, etc. of the physical data model andaugment data model, it is determined whether and to what degree some orall of the augment visual model is behind (i.e. visually obstructed by)some or all of the physical visual model. Portions of the augment visualmodel that are occluded by the physical visual model may be identifiedin some fashion, or the occlusion may be otherwise noted for furtherconsideration in subsequent steps.

Still with reference to FIG. 8, physical visual content is generated727, using as source material the physical visual model. Typically,though not necessarily, the physical visual content may betwo-dimensional so as to be adapted for output using a two-dimensionaldisplay (even though the physical visual model may be three-dimensionalor of some other form).

Also typically though not necessarily, the physical visual content maybe generated within a processor, such as one having suitable executableinstructions instantiated thereon. For example, a graphical engine mightrender a frame (or series of frames) showing the contents of thephysical visual model in visible form as would be seen from the viewingposition and bounded within some field of view. However, otherarrangements may be equally suitable.

Augment visual content also is generated 828, using as source materialthe augment visual model and the occlusion. Thus, at least a portion ofthe augment visual model is configured as data suitable for visualoutput, with occlusions of the augment visual model being excluded fromvisual output.

Physical visual content is outputted 830 to the viewing position. Forexample, for a head mounted display utilizing one or more LED screens,the physical visual content may be displayed as an arrangement of pixelsforming a visible image. Such a visible image may depict the contents ofthe physical visual model as seen from the viewing position and boundedwithin a field of view (e.g. a field of view substantially correspondingwith the edges of the LED screen). However, this is an example only, andother arrangements may be equally suitable.

As previously noted, the example method of FIG. 8 is presented asspecific to use with a virtually (rather than an optically) transparenthead mounted display. Thus, for the particular example of FIG. 8 thedisplay may not be optically transparent and indeed may be entirelyopaque, but physical visual content is outputted thereto in a manner atleast substantially corresponding to how physical visual content may bearranged if the head mounted display were optically transparent. Morecolloquially, the head mounted display actively outputs physical visualcontent so as to provide the visual appearance of being opticaltransparent. This is distinct from the arrangement in FIG. 7, whereinoutput of physical visual content was passive, relying on the opticaltransparency of the head mounted display.

The augment visual content also is outputted 832 to the viewing positionwith the head mounted display. As with step 830, for a head mounteddisplay utilizing one or more LED screens, the augment visual contentmay be displayed as an arrangement of pixels forming a visible image.Such a visible image may depict the contents of the augment visual modelas seen from the viewing position and bounded within a field of view(e.g. a field of view substantially corresponding with the edges of theLED screen). However, this is an example only, and other arrangementsmay be equally suitable.

Although FIG. 8 shows the method therein as being complete followingstep 832, other steps, other functions, etc. may be incorporated intothe method, and/or other methods may be executed in combination with themethod. For example, as noted with regard to FIG. 6 and FIG. 7 for atleast certain embodiments at least some portion of the method mayrepeat. With such an arrangement, each repetition may yield a new stillimage displayed on the head mounted display, so that the physicalaugment content and/or visual augment content as visible to a viewer maybe dynamic in time. A sufficiently rapid rate of update may produce theappearance of continuous change and motion in the physical and/or visualaugment content, presenting at least the impression of a smoothlyvariable, interactive, responsive visual experience for the viewer.

At this point, some additional comments regarding certain features ofvarious embodiments may be illuminating. As noted, certain examplesdescribed and illustrated thus far have focused on occlusion as anexample feature to be supported through integrating physical andnotional content. Although individual features have been explained thusfar, a more complete understanding may benefit from consideration ofoverall functionality.

According to various embodiments some property of two different forms ofcontent is integrated across the two forms of content. In certainexamples presented thus far herein, the property in question isocclusion, the tendency of one object or feature to be “in the way” ofanother object or feature, so as to obscure some or all of more distantfeatures. Similarly, in certain examples presented thus far herein, oneform of content is notional (virtual reality, augmented reality, etc.)while the other form of content is physical (“real world” objects andfeatures).

As has been described, properties of one data model do not necessarilyapply to another data model. By default, an augmented reality objectdisplayed on a head mounted display may appear overlaid onto (e.g.projected over) a physical object even if the augmented reality objectis nominally more distant than the physical object (for example due tothe manner by which the notional object is displayed); that is, the moredistant augmented reality object will not be occluded by the physicalobject. This may be the case even though a distant physical object maybe occluded by a closer physical object, and/or a distant augmentedreality object may be occluded by a closer augmented reality object.Occlusion of distant objects by near objects applies within both theaugmented reality and physical models, but does not necessarily applybetween the two models.

By contrast, according to various embodiments occlusion (or some otherproperty) is made to apply consistently between the two models, suchthat any entity in one model may (if desired) be made to occlude anyentity in the other model. Occlusion thus may be said to be integratedas a property across the two data models.

It may be worth drawing a distinction between certain embodiments and anapproach that might be considered as in some manner superficiallysimilar: so-called “green screen” imaging.

In green screen imaging, a single narrow band of color (typically greenor blue) is defined as identified within an image and replaced withother content. Considering as an example a weather forecaster standingin front of a green screen, the green may be selectively identified bycolor and replaced with a satellite image showing weather over an area.The visual effect is that the weather forecaster remains visible in theforeground, blocking a portion of the satellite image behind him or her.It might be argued that this in some sense constitutes integration, inthat the weather forecaster—a close physical entity—blocks sight to aportion of a projected satellite image (which might be compared, thoughperhaps not correctly, to a notional entity). However, the approach andfunctionality of green screen differs from that of various embodiments.

First, green screen does not distinguish based on distance, and in factmakes no consideration of distance. In various embodiments, occlusion ofdistant notional entities by close physical entities is brought about.By contrast, the effect in a green screen is not produced because theweather forecaster (to continue the example above) is in front of thescreen, but because the weather forecaster is a different color than thescreen. Distance is of no consequence; if the forecaster were to wear ajacket of the proper shade of green, the area of the jacket in imageswould be replaced with the satellite image just as the green screen is,even though the green screen is more distant than the forecaster and thejacket is not.

That is, green screen considers color, not on distance, and distance isnot a factor. Green screen thus does not integrate occlusion betweendata models, since occlusion depends at least in part on distance. Also,green screen is thus limited only to considering objects in a verynarrow color range, where embodiments are not necessarily color limited.(In principle an embodiment might be made that, for whatever reason,only considers entities of a certain color for purposes of determiningocclusion, but this is not required.)

Second, as may be implicit in the first point, green screen depends onhaving a physical object representing the region onto which an image isprojected, and in front of which some foreground entity may be leftvisible. The “screen” in green screen is a physical requirement,necessary for the green screen visual effect; a physical object(specifically, a physical object in the proper shade of green) must bedirectly associated with any viewed image. The viewed image in a greenscreen system thus is not necessarily even similar to a notional entity;a notional entity may have no physical substance, and need not berepresented by a physical object.

As a result of the features noted in the first and second points above,green screen typically is limited to being a studio effect. Green screenis poorly suited for “free range” applications, such as those associatedwith head mounted displays and other portable devices, since in generalenvironments may not be relied upon to have suitable physical objects insuitable shades of green present when and where necessary.

Also, as may be understood from the first and second points above, greenscreen relies on physical occlusion of one physical object by another,namely, the forecaster occluding the green screen. In practice greenscreen occlusion is not integrated across a physical data model and anotional model.

Third, green screen does not support change in apparent relativedistance. That is, the weather forecaster must always appear in front ofthe satellite image. A change in position cannot put the forecastermidway through the satellite image, with (say) only one arm extendingthrough the satellite image. A change in position likewise cannot enablethe forecaster to pass through the satellite image. Objects in front ofa green screen are fully in front of the screen, or fully behind thescreen; while in principle multiple green screens could be arranged sothat some such are in front of a target object and others behind thetarget object, each such green screen is itself fully in front of orfully behind the target object.

This behavior of green screens is at least in part due the fact that thegreen screen is required to be a physical object, physically present, asnoted above. A forecaster cannot walk through a satellite image becausehe or she could not walk through the required physical green screen.

This is entirely different from various embodiments described herein. Invarious embodiments, a physical entity may pass through a notionalentity or vice versa, and a physical entity may extend partway out of anotional entity or vice versa. Thus, what is occluded and what is notoccluded varies, based on the distances as those distances change.

In sum, embodiments provide functionality not present in green screen,and operates in a different manner and along different principles.

As a further comment, it is pointed out that where green screen islimited only to supporting some limited visual effects that maysuperficially resemble (but does not equate to) integrated occlusionbetween notional and physical data models, embodiments are not limitedonly to integrating occlusion among data models.

Rather, embodiments may enable substantially any sensory property to beintegrated among data models.

Now with reference to FIG. 9, therein and in certain subsequent figuresanother example of a property integrated among notional and physicaldata models is provided therein. More particularly, beginning in FIG. 9the property of shadowing from light sources is shown to be integratedamong notional and physical entities and light sources.

In FIG. 9, a physical entity 908 is shown therein represented by a cube,and a notional entity 910 represented by a tetrahedron. A flat plane 942of limited dimension is shown for purposes of illustrating projectedshadows, though this is a convenience done for clarity and otherarrangements may be equally suitable.

Also in FIG. 9, a physical light source 944 is shown therein representedby a wireframe sphere. The physical light source 944 may besubstantially any “real life” object or feature that emits light, suchas an LED, incandescent bulb, etc. In addition, a shadow 948 is cast bythe physical entity 908 from the physical light source 944. Radial lines946 extending outward from the physical light source 944 past thevertices of the physical entity 908 indicate the shape of the shadow948; the radial lines 946 are visual guides only, and are notnecessarily present.

The shadow 948 in FIG. 9 may be referred to herein more specifically asa physical-physical shadow 948, wherein the use of the word “physical”before the dash references a physical light source 944, and the use ofthe word “physical” after the dash references a physical entity 908.Thus, the physical-physical shadow 948 in FIG. 9 is the product of aphysical light source and a physical entity.

As may also be observed from FIG. 9, no shadow is shown to be cast bythe notional entity 910 from the physical light source 944.

Turning now to FIG. 10, a physical entity 1008 is shown thereinrepresented by a cube, and a notional entity 1010 represented by atetrahedron. A notional light source 1050 is shown therein representedby a wireframe cylinder. The notional light source 1050 may be anynotional construct that emits light, or that behaves for purposes ofother notional entities as though emitting light. For example, notionallight sources may include but are not limited to a virtual or augmentedreality sun, moon, torch, lamp, etc. In addition, a shadow 1056 is castby the notional entity 1010 from the notional light source 1050, asvisible on the plane 1042. Radial lines 1053 extending outward from thenotional light source 1050 past the vertices of the notional entity 1010indicate the shape of the shadow 1056; as noted with regard to FIG. 9,the radial lines 1053 in FIG. 10 also are visual guides only, and arenot necessarily present.

The shadow 1056 in FIG. 10 may be referred to herein more specificallyas a notional-notional shadow 1056, wherein the use of the word“notional” before the dash references a notional light source 1050, andthe use of the word “notional” after the dash references a notionalentity 1010. Thus, the notional-notional shadow 1056 in FIG. 10 is theproduct of a notional light source and a notional entity.

As may again be observed from FIG. 10, no shadow is shown to be cast bythe physical entity 1008 from the notional light source 1050.

Now with reference to FIG. 11, a physical entity 1108 is shown thereinrepresented by a cube, and a notional entity 1110 represented by atetrahedron. A physical light source 1144 is shown represented by awireframe sphere, and a notional light source 1150 represented by awireframe cylinder. As may be observed, a physical-physical shadow 1148is cast by the physical entity 1108 from the physical light source 1144as visible on the plane 1142, and a notional-notional shadow 1156 iscast by the notional entity 1110 from the notional light source 1150 asvisible on the plane 1142.

As may also be observed, no shadow is shown cast by the physical entity1108 from the notional light source 1150, nor a shadow cast by thenotional entity 1110 from the physical light source 1144.

To summarize with respect to FIG. 9 through FIG. 11, it may be observedthat a physical light source may cast a shadow from a physical object,and a notional light source may cast a shadow from a notional object.However, in the examples of FIG. 9 through FIG. 11 a physical lightsource does not cast a shadow from a notional object, nor a notionallight source from a physical object. As a more concrete example, areal-world lamp may not cast shadows from virtual reality objectsdepicted on a head mounted display, and likewise a virtual reality lightsource as depicted by a head mounted display may not cast shadows fromphysical objects.

This may be understood in that shadowing in a physical environment is anautomatic physical process, a result of the natural behavior of light.However, where no physical object is present (e.g. for the case of anotional object), physical shadowing may not occur. Conversely,shadowing in a notional environment typically is a computationallydetermined process, with shadows produced based on constructedalgorithms that apply only within the notional environment. Thus,shadowing from objects not part of the notional environment (e.g. aphysical object) also may not occur.

Even though both the physical and notional content both exhibitshadowing from light sources, the two data models are not integratedwith regard to the property of shadowing.

However, in various embodiments a sensory property such as a visualfeature effect may be found present in one such data model, and a secondanalogous sensory property determined for the other data model, thatsecond sensory property then being outputted to a viewer e.g. byapplying the second sensory property to the notional sensory content tobe outputted therewith.

More concretely with regard to the examples of FIG. 9 through FIG. 11,although typically (without integration through various embodiments)physical light sources generate shadows only from physical objects, andnotional light sources generate shadows only from notional objects,shadowing may be integrated between the two data models.

Now with reference to FIG. 12, an example arrangement of integratingshadowing between physical and notional data models according to variousembodiments is shown therein.

In FIG. 12, a physical entity 1208 is shown therein represented by acube, and a notional entity 1210 represented by a tetrahedron. Aphysical light source 1244 is shown represented by a wireframe sphere. Aphysical-physical shadow 1248 is cast by the physical entity 1208 fromthe physical light source 1244 as visible on the plane 1242, with radiallines 1246 marking the vertices thereof.

In addition, as may be observed a physical-notional shadow 1258 is alsocast by the notional entity 1210 from the physical light source 1244 asvisible on the plane 1242, with radial lines 1257 marking the verticesthereof.

The arrangement in FIG. 12 shows an arrangement as may be viewed throughthe use of various embodiments. A description of how such results may beimplemented is presented subsequently herein; initially, with regard toFIG. 12 (and likewise FIG. 13 and FIG. 14) an appearance of an exampleembodiment is presented, for purposes of clarity.

In FIG. 13, a physical entity 1308 is shown therein represented by acube, and a notional entity 1310 represented by a tetrahedron. Anotional light source 1350 is shown represented by a wireframe cylinder.A notional-notional shadow 1356 is cast by the notional entity 1310 fromthe notional light source 1350 as visible on the plane 1342, with radiallines 1353 marking the vertices thereof.

In addition, as may be observed a notional-physical shadow 1362 is alsocast by the physical entity 1308 from the notional light source 1350 asvisible on the plane 1342, with radial lines 13607 marking the verticesthereof.

Turning to FIG. 14, a physical entity 1408 is shown therein representedby a cube, and a notional entity 1410 represented by a tetrahedron. Aphysical light source 1444 is shown represented by a wireframe sphere,and a notional light source 1450 represented by a wireframe cylinder.

A physical-physical shadow 1448 is cast by the physical entity 4108 fromthe physical light source 1444 as visible on the plane 1442. Aphysical-notional shadow 1458 is cast by the notional entity from thephysical light source 1444 as visible on the plane 1442. Anotional-notional shadow 1456 is cast by the notional entity 1410 fromthe notional light source 1450 as visible on the plane 1442. Anotional-physical shadow 1462 is cast by the physical entity 1408 fromthe notional light source 1450 as visible on the plane 1442.

Thus, as is visible in the example of FIG. 14, insofar as a personviewing the combined physical and notional visual content is concerned,shadows appear from both physical and notional entities illuminated byphysical light sources, and shadows also appear from both physical andnotional entities illuminated by notional light sources. With regard tothe property of shadowing, the physical and notional visual content thusis integrated.

For purposes of explanation, terminology in certain previous exampleshas referred to a notional or a physical light source, and a notional ora physical entity. Thus, a notional-physical shadow would be one cast bya physical entity from a notional light source (the term being of theform “light-object”). However, a further possibility also may beconsidered, namely whether the surface onto which a shadow is castlikewise is either physical or notional. Given a notional surface, eventhe combination of a physical light source and a physical object may notinherently result in a projected shadow without active integration. Morecolloquially, a real light and a real object still will notautomatically produce a shadow on a virtual object.

Typically, though not necessarily, a shadow cast on a notional surfacemay be a notional entity regardless of whether the light and/or theshadowing entity are themselves physical or notional. That is, thenotional surface “isn't there” for a shadow to be generated thereon by aphysical light source and/or a physical entity.

Thus although in certain instances entities are referred to herein withtwo terms, e.g. “notional-physical” for a shadow cast by a physicalentity from a notional light source, a broader approach may be to referto entities with three terms, e.g. “notional-physical-notional” (of theorder “light-object-surface” for a shadow cast by a physical entity froma notional light source onto a notional surface.

Given such an arrangement, a total of eight different classes of shadowmay be considered, abbreviated PPP, PPN, PNP, PNN, NPP, NPN, NNP, andNNN. PPP shadows—produced by physical lights, entities, andsurfaces—occur naturally due to the physics of light. NNNshadows—produced by notional lights, entities, and surfaces—occurentirely within notional environments, e.g. being determinedcomputationally within and as part of a virtual reality. For theremaining six classes, shadows are applied through integration accordingto various embodiments. However, for those six classes—PPN, PNP, PNN,NPP, NPN, and NNP—shadows typically are themselves notional entities (asis a shadow for NNN, though in the case of NNN this may be less notablebecause all of the entities are notional). More colloquially, shadowsmust be “artificial”, such as virtual or augmented reality objects,because physical lights, objects, and surfaces may not naturally exhibitshadows when combined with virtual/augment lights, objects, andsurfaces. It is through integration according to various embodimentsthat such shadows are provided.

More regarding a specific example of such notional shadowing, and withconsideration of a virtual surface, is said later herein beginning withFIG. 23.

Moving on, with reference to FIG. 15, therein is shown an example methodwherein physical and notional sensory content may be integrated. Theexample in FIG. 15 is not specific to any particular property, nor toany particular sensory content, and embodiments likewise are not limitedwith regard to property or sensory content. However, it is noted thatthe method shown in FIG. 15 bears at least some similarity to the methodshown in FIG. 6, wherein a property of occlusion for visual content isconsidered; likewise, FIG. 16 (described subsequently) presents a methodwherein a property of shadowing for visual content also is considered.Occlusion and shadowing are examples of properties, and visual contentis an example of sensory content, but embodiments are not limited onlythereto.

In FIG. 15, a reference position is established 1520. The referenceposition serves as a positional baseline with regard to integratingsensory content for certain embodiments. For example, for purposes ofvisual content, the reference position may be a viewing position (e.g.possibly corresponding with a known or anticipated position of aperson's eye), while for auditory content the reference position may bea listening position (e.g. possibly corresponding with a known oranticipated position of a person's ear), etc.

A physical data model is established 1522. The physical data model is anassembly of data that describes at least one parameter of at least onephysical entity. Typically the physical data model addresses to at leastsome degree factors that may affect the sensory content and/or propertyunder consideration. That is, for visual content, the physical datamodel may include and/or represent visual information regarding physicalentities. Likewise, shadowing as a property the physical data model mayinclude and/or represent information regarding sources of illumination,the position and size of physical entities, the degree of transparency(if any) of physical entities, etc.

Continuing in FIG. 15, a notional data model is established 1524. Thenotional data model is an assembly of data that describes at least oneparameter of at least one notional entity. As with the physical datamodel, typically the notional data model addresses to at least somedegree factors that may affect the sensory content and/or property underconsideration.

The physical and/or notional data models may be dynamic in time.

A first sensory property is established 1525 in the physical data modelor the notional data model. Embodiments are not limited with regard tothe sensory property, and substantially any effect, phenomenon,behavior, etc. that may be sensed may be suitable. Previously withregard to FIG. 1 through FIG. 8, occlusion has been described herein asan example property. However, many other properties may be establishedas the first sensory property.

For example, as described with regard to FIG. 9 through FIG. 14 herein,shadowing may serve as a first sensory property. Similarly, otherproperties such as illumination may serve as a first property. (Withcomparison to FIG. 9 through FIG. 14, just as typically a physical lightsource may not naturally produce an appearance of shadowing for anotional object and a notional light source may not produce anappearance of shadowing for a physical object, likewise a physical lightsource naturally may not produce an appearance of illumination upon anotional object and a notional light source may not produce anappearance of illumination upon a physical object.)

Suitable properties may include, but are not limited to, opticalproperties such as diffraction, diffusion, focus (e.g. being “out offocus”), “glory”, haloing, lens flare, and reflection.

Suitable properties may also be volumetric in nature, affecting a regionof space (whether physical or notional space) rather than a moreconcrete entity such as an object. For example, the presence of smokewithin a volume of the physical or notional environment (and representedwithin the physical or notional model) may be suitable as a firstsensory property. With regard particularly to smoke, such a property mayrepresent a reduction of visual detail, a decrease in color saturationand/or an overlay of gray, decreased illumination, etc. Indeed, suchmore abstract notions—e.g. reduction of visual detail—may themselvesserve as first sensory properties, without necessarily correlatingdirectly with any concrete phenomenon. Some, though not necessarily all,volumetric properties may be spatially varying, e.g. having an effectthat is a function of the distance from the reference point (forexample, the effects of fog on visibility tend to become more pronouncedat greater distances).

Other suitable volumetric properties may include, but are not limitedto, ash, debris, dust, fog, gas, hail, heat distortion, insects, leaves,mist, rain, sleet, snow, spray, and steam. Volumetric properties maytake the form of falling, flying, and/or suspended elements, but also(as noted above) may be abstractions without any concrete substance.

It is emphasized that the first sensory property may be established 1525from, and/or present in, either the physical data model (and typicallythough not necessarily the physical environment) or the notional datamodel (and typically though not necessarily the notional environment).

In addition, it is noted that if a first sensory property ispre-determined, step 1525 may be implicit, rather than necessarily beingperformed as an active part of a method. For example, for an embodimentswherein a single, specific property is to be addressed—such as occlusionin FIG. 6 through FIG. 8—that property may be considered to beestablished without specifically referencing such a step.

Moving on in FIG. 15, a second sensory property is determined 1526 forthe notional model, the second visual effect at least substantiallycorresponding with the first sensory property. That is, if the firstsensory property is shadowing (whether in the physical data model ornotional data model), the second sensory property at least substantiallycorresponds to shadowing as well.

Continuing to consider shadowing as an example, and to describe therelationship in more concrete terms, if the first sensory property isshadowing of physical objects from physical light sources, then thesecond sensory property may represent shadowing of notional objects fromphysical light sources. The first sensory property (physical-physicalshadowing) already exists, in this example within the physical datamodel (typically as a representation of shadowing within the physicalenvironment); the second sensory property (physical-notional shadowing)is then determined 1526 so as to substantially correspond to the firstsensory property, e.g. so as to present the appearance of shadows fornotional objects from physical light sources that are similar or evenidentical to shadows for physical objects from physical light sources.

Possibly, though not necessarily, the method shown in FIG. 15 and/orcertain steps thereof, including but not limited to step 1526, may beperformed multiple times, determining 1526 corresponding second sensoryproperties for several different first sensory properties. One exampleof such an arrangement is a “two way” correspondence: physical-notionalshadows may be generated that correspond with physical-physical shadows,and notional-physical shadows likewise may be generated that correspondwith notional-notional shadows. This result of such an arrangement maybe similar to that shown in FIG. 14, with both notional-physical shadow1462 visible therein (e.g. as if being determined to correspond withnotional-notional shadow 1456 such that both shadows 1456 and 1462appear to cast from notional light source 1450) and physical-notionalshadow 1458 visible therein (e.g. as if being determined to correspondwith physical-physical shadow 1448 such that both shadows 1448 and 1458appear to be cast from physical light source 1444). Thus, a property—inthis example, shadowing—may be integrated in both directions between twodata models.

However, certain embodiments may integrate only one first property inone “direction”. With reference to occlusion as shown in FIG. 1 throughFIG. 5B, a second property of occlusion for the notional data model isdetermined as corresponding with occlusion in the physical data model,but not the other way around.

More colloquially, notional objects are made to appear occluded byphysical objects, but no particular effort is made to occlude physicalobjects appear occluded by notional objects. As may be understood, forat least certain circumstances such as the use of certain head mounteddisplays, occlusion of physical objects by notional objects displayed onthe head mounted display is a natural effect, manifesting without activeintervention.

However, regardless of whether such properties manifest naturally ornot, embodiments do not require that sensory properties be integrated“in both directions” (though such also is not prohibited, as may be seenwith regard to FIG. 14).

Again with reference to step 1526 in FIG. 15, it is noted thatregardless of whether the first property under consideration isoriginally present in the physical data model or the notional datamodel, the second sensory property is determined for notional datamodel. Modifications, additions, subtractions, etc. in the appearance ofa notional environment typically may be executed in a processor, such asone with executable instructions instantiated thereon, wheremodifications, additions, subtractions, etc. in a physical environmenttypically may not be. Again more colloquially, a virtual or augmentedreality world may be changed in a processor, where the real worldtypically cannot.

A distinction may be made for purposes of clarity and understandingbetween which data model the second sensory property is addressed to,and which data model the second sensory property is aimed for. Thesecond sensory property is always addressed to the notional data model,in so far as (as noted) the notional data model is and/or represents anotional environment, that may be readily modified within a processor.However, the second sensory property may be aimed for either thephysical data model or the notional data model. That is, the secondsensory property may give the appearance of an effect within thephysical world, or within the notional world. As a more concreteexample, a second sensory property may contribute to producing theappearance of fog in the physical world, i.e. obscuring physical objects(as perceived by a viewer), and/or may contribute to producing theappearance of fog in the notional world, i.e. obscuring virtual objects,augmented reality objects, etc. (again as perceived by the viewer).

The determination 1526 of the corresponding second sensory propertytypically is carried out in a processor, such as a processor havingexecutable instructions instantiated thereon. Embodiments are notlimited to the manner in which the determination 1526 is made. Forpurposes of shadowing, the determination might consider the position,size, orientation, optical properties, etc. of objects, light sources,etc. as represented in the physical data model and/or the notional datamodel, and determine a second sensory property substantiallycorresponding to the first sensory property (shadowing) based on opticalprinciples, geometric analysis, etc. However, this is an example only,and other arrangements may be equally suitable.

Substantial correspondence between properties may be considered withregard to physical, functional, and/or other features. For example, fora first sensory property relating to the presence of fog within thephysical or notional data models, the second sensory property mayinclude modeling of the presence, distribution, size, behavior, etc. ofwater droplets or similar, thus corresponding physically with the firstsensory property. However, the second sensory property may also includemodeling of a decrease in visibility over distance as a feature of fog,with no consideration given to the physical nature of fog. Otherarrangements also may be equally suitable.

Continuing in FIG. 15, notional sensory content is generated 1528 fromthe sensory data model with the second sensory property applied thereto.Thus, shadows determined as a second sensory property may be appliedwhen generating notional sensory content. Likewise fog, smoke, fallingleaves, being “out of focus”, etc. may be applied when generatingnotional sensory content.

Physical sensory content is outputted 1530 to the reference position. Asnoted above with regard to FIG. 6, this may be a passive step, such aswhen an optical see-through display passes visual appearances of aphysical environment therethrough. Thus, step 1530 may for at leastcertain embodiments be optional. However, as also noted with regard toFIG. 6, for other embodiments outputting physical sensory content 1530may be an active step, as well.

The notional sensory content is also outputted 1532 to the referenceposition. As noted with regard to step 1528 the second sensory propertyis applied thereto, the second sensory property at least substantiallycorresponding with the first sensory property. For example, for asee-through head mounted display, and considering shadows generated byphysical light sources and physical objects to be the first sensoryproperty, and shadows generated by physical light sources and notionalobjects to be the substantially corresponding second sensory property,as viewed with the head mounted display notional objects may exhibit theappearance of having shadows as cast from physical light sources.

Thus, in colloquial terms, some visual, auditory, or other sensoryproperty present in one of the physical and notional worlds may be madeto appear to be present in the other of the physical and notionalworlds. Where shadowing from physical light sources is present forphysical objects, notional objects also may be made to exhibit theappearance of shadowing from those physical light sources; whereinshadowing from notional light sources is present for notional objects,physical objects also may be made to exhibit the appearance of shadowingfrom those notional light sources.

In addition, with reference to step 1528, in the example embodiment ofFIG. 15 the occlusion is applied in generating content from the notionaldata model. However, for at least certain embodiments it may be equallysuitable to apply the occlusion in generating content from the physicaldata model, and/or in generating content from both models. The occlusionalso may be used to generate occlusion content entirely separately, anddelivered as a third “stream” of content. Embodiments are not limited tothe manner in which the occlusion is made to appear as visual content.However, since at least certain embodiments may utilize an opticalsee-through display, wherein no physical visual content is generated(light reflecting from physical objects in such case being passivelytransmitted through the optical see-through display), for purposes ofclarity the occlusion is described chiefly herein as being applied tocontent generated from the notional data model (i.e. because content isnot necessarily generated from the physical data model).

Furthermore, although shadowing and occlusion are described herein asexamples (in FIG. 15 and FIG. 6 respectively), as noted alreadyembodiments are not limited only to such properties. Through variousembodiments, reflective physical objects may be made to exhibit theappearance of reflections of notional objects, or vice versa; notionalobjects may be made to exhibit the appearance of being obscured by thepresence of physical fog, or vice versa.

As has been noted, occlusion and shadowing may be so integrated amongphysical and notional data models through various embodiments. Inaddition, illumination, optical effects such as diffraction, diffusion,focus, glory, haloing, lens flare, and reflection, and volumetriceffects such as ash, debris, dust, fog, gas, hail, heat distortion,insects, leaves, mist, rain, sleet, smoke, snow, spray, and steam may beintegrated among physical and notional data models through variousembodiments.

As an example of a specific application of such examples, consider anarrangement wherein a person is wearing a see-through head mounteddisplay while playing a mixed-reality game (i.e. a game wherein changesto the appearance of the physical environment, additions, subtractions,etc. are made via virtual reality, augmented reality, etc.). If, forexample, a game-generated dragon were to fly overhead, a shadow (or atleast visual content substantially corresponding with shadows) of thatdragon could be overlaid onto the appearance of the physical world. Ifreal-world fog were present in the vicinity of the player, artificialgame elements (e.g. the aforementioned dragon) could be obscured throughthe application of an appearance of fog within the artificial gameworld. Thus, through various embodiments, the two “worlds” may beintegrated in their appearance.

As also noted previously, embodiments are not limited only to visualsensory properties. To continue the example immediately above, if theplayer is in an enclosed space in the physical world, such as an alley,such that sounds made within that space generate audible echoes,substantially corresponding echoes may be generated and outputted theplayer (e.g. through audio outputs on the head mounted display) for thein-game sounds as well.

Again with reference to FIG. 15, the example method shown therein may beconsidered to be a broad method, applicable to substantially any sensoryproperty or properties, and substantially any sense; the examples ofocclusion and shadowing as presented previously herein may be consideredto be more specific examples, but embodiments are not limited onlythereto.

Now with reference to FIG. 16A and FIG. 16B, another example arrangementis presented whereby physical and notional sensory content may beintegrated, again with reference to shadowing as an example. However,where the example method in FIG. 15 showed a general method, the examplemethod in FIG. 16A and FIG. 16B provides an example of more particularapproaches for certain steps within the method, and also addresses themethod to use with a specific form of apparatus, namely an opticallytransparent head mounted display adapted for augmented reality output.It is emphasized that embodiments are not limited only to such anapparatus, and that other arrangements may be equally suitable.

In FIG. 16A, a viewing position is established 1620 for a head mounteddisplay. For example, given a head mounted display in the form of a pairof glasses, a viewing position may be established 1620 at or near theanticipated position of a wearer's eye and/or eyes.

Continuing in FIG. 16A, camera images are obtained 1622A, in thespecific example of FIG. 16A a color camera image and a depth cameraimage. A color image, such as an RGB color image (though other imagesmay be equally suitable) provides information regarding illumination,color, transparency, etc. of a physical environment. A depth cameraimage, such as an infrared time-of-flight depth image (though againother images may be equally suitable) provides information regardingthree dimensional shape, position, and orientation of the physicalenvironment. While an color camera image and depth camera image may besuitable for certain embodiments, this is an example only, and otherarrangements also may be equally suitable.

The camera images are received 1622B in a processor, such as a processordisposed on or in the head mounted display.

A physical visual model is generated 1622C with the processor from thecamera images and/or data therein. The physical visual model typically(though not necessarily) represents at least a portion of the physicalreality environment proximate the head mounted display. For the specificexample in FIG. 16A, a three dimensional visual model of a physicalenvironment may be generated by combining information regarding color,surface texture, transparency, etc. from the color camera image withinformation regarding shape, orientation, distance, etc. from the depthcamera image. However, other arrangements also may be equally suitable.The physical visual model may be dynamic over time.

Steps 1622A, 1622B, and 1622C may be considered collectively as being atleast somewhat comparable to step 1522 in FIG. 15 wherein a physicalsensory model is established, though the arrangement in FIG. 16Apresents a more specific example as opposed to the more broad generalarrangement in FIG. 15.

Still with reference to FIG. 16A, an augment visual model is established1624, the augment visual model representing an augmented realityenvironment that is typically though not necessarily proximate the headmounted display. The augment visual model may be dynamic over time.

The geometry of one or more light sources in the physical model isestablished 1625A. The geometry of light sources refers to where thoselight sources are in space, where the light sources are aimed, and soforth. As noted above, for the particular example in FIG. 16A theposition, orientation, size, etc. of physical light sources may bedetermined for example through analysis of data from the depth cameraimage and/or color camera image, though information from other sourcesalso may be utilized instead or in addition.

The geometry of shadowing entities in the physical visual model isestablished 1625B. The geometry of shadowing entities refers to whereobjects adapted to cast shadows (including but not limited to solidopaque physical objects) appear within the physical visual model, thesize and shape of such objects, etc. As noted above, for the particularexample in FIG. 16A the position, orientation, size, etc. of physicalshadowing entities may be determined for example through analysis ofdata from the depth camera image and/or color camera image, thoughinformation other sources also may be utilized instead or in addition.

The geometry of shadowing surfaces in the physical visual model isestablished 1625C. The geometry of shadowing surfaces refers to theposition, shape, orientation, etc. of the surface or surfaces on which ashadow is cast in the physical visual model. For example, in FIG. 9through FIG. 14 shadows therein are projected onto a flat plane.However, in practice shadows may be projected onto more complexgeometry, thus for at least some embodiments the shadowing surfacegeometry may be significant. However, not all embodiments necessarilywill benefit from or require data regarding shadowing surface geometryin order to determine corresponding shadows (as in subsequent stepsherein), and other arrangements may be equally suitable. Again as notedabove, for the particular example in FIG. 16A the position, orientation,size, etc. of physical shadowing surfaces may be determined for examplethrough analysis of data from the depth camera image and/or color cameraimage, though information other sources also may be utilized instead orin addition.

Continuing in FIG. 16A, the quality of shadows in the physical visualmodel is established 1625D. The quality of shadows refers to suchfactors as darkness, contrast between lit and shadowed regions, color,diffusion, etc. This may be considered alternately in terms of qualitiesof the light sources themselves, e.g. the color of the light emitted,the brightness, etc. Either or both approaches may be suitable, or someother approach. For purposes of the example in FIG. 16A, the shadowquality is established 1625D, though other arrangements may be equallysuitable.

Steps 1625A, 1625B, 1626C, and 1625D may be considered collectively asbeing at least somewhat comparable to step 1625 in FIG. 15 wherein afirst sensory property is established in the physical or notional datamodel, though the arrangement in FIG. 16A presents a more specificexample for a first sensory property (shadowing in the physical modelfrom physical light sources) as opposed to the more broad generalarrangement in FIG. 15.

Moving on in FIG. 16A, the geometry for one or more light sources forthe augment visual model at least substantially corresponding with thephysical light sources is established 1626A. For example, the position,size, orientation, etc. of a light source within the augment visualmodel adapted to contribute to shadows for the augment entities thatsubstantially correspond with the shadows already established (in steps1625A through 1625D) for the physical entities in the physical visualmodel may be established 1626A. This may be determined for examplethrough analysis of the information regarding the geometry of the lightsources in the physical model as established 1625A previously herein,though other arrangements may be equally suitable.

The geometry for shadowing entities within the augment visual model isestablished 1626B. Typically though not necessarily this may beestablished through analysis of the augment visual model, e.g. whichaugment objects are in positions as to cast shadows, how those augmentobjects are positioned and oriented, etc. However other arrangementsalso may be equally suitable.

The geometry of shadowing surfaces within the augment visual model isestablished 1626C. Again, typically though not necessarily this may beestablished through analysis of the augment visual model, e.g. whichaugment objects are in positions as to receive shadows projectedthereon, how those augment objects are positioned and oriented, etc.However other arrangements also may be equally suitable.

Now with reference to FIG. 16B, the geometry of shadows is determined1626D, for shadows of augment objects from physical light sources atleast substantially corresponding with shadows of physical objects fromphysical light sources. (That is, physical-notional shadows, or for thisparticular example physical-augment shadows.) The geometry of shadowsrefers to the shape, size, position, etc. of shadows of augment objectsas are determined to be projected from physical light sources (or dataconstructs corresponding to physical light sources). Typically thoughnot necessarily the geometry of physical-augment shadows may bedetermined computationally, through analysis of data regarding lightsources, shadowing objects, etc. in view of properties of lightpropagation.

The quality of substantially corresponding shadows is determined 1626E.As with the quality of physical shadows established in step 1625D,quality refers to factors such as color, darkness, contrast, etc. Againas noted with regard to step 1625D such factors may be considered withregard to light sources in addition to and/or instead of with regard tothe shadows themselves.

Steps 1626A, 1626B, 1626C, 1626D, and 1626E may be consideredcollectively as being at least somewhat comparable to step 1626 in FIG.15 wherein a sensory property is established with respect to thenotional data model, though the arrangement in FIG. 16A and FIG. 16Bpresents a more specific example for a second sensory property(shadowing in the notional model as if from physical light sources) asopposed to the more broad general arrangement in FIG. 15.

It is emphasized that although FIG. 16A and FIG. 16B show particulararrangements of method steps for (in the example given) establishingshadowing within a physical data model, determining substantiallycorresponding shadowing for an augment model, and delivering as outputthat substantially corresponding shadowing, embodiments are not limitedonly to these steps in integrating shadowing between physical andnotional models. Likewise, in integrating other sensory properties,embodiments are not limited only to the steps shown in FIG. 16A and FIG.16B (and indeed depending on the particular sensory property, suitablesteps may vary considerably).

Still with reference to FIG. 16B, augment visual content is generated1628, using as source material the augment visual model and the shadowsas determined in steps 1626A through 1626E. Thus, at least a portion ofthe augment visual model is configured as data suitable for visualoutput, with those portions determined for shadowing (e.g. tagged areas,masks, etc., though other arrangements may be equally suitable) beingincluded therein. Typically, though not necessarily, the augment visualcontent may be two-dimensional so as to be adapted for output using atwo-dimensional display (even though the augment visual model and/or theindication(s) of occlusion may be three-dimensional or of some otherform).

As previously noted, the example method of FIG. 16A and FIG. 16B ispresented as specific to use with an optically transparent head mounteddisplay. Thus physical visual content may be considered to pass throughthe display to the viewing position without an active step in theexample method. This passive optical transmission may be in some senseanalogous to step 1530 in FIG. 15, though as noted the method does notrequire action.

Continuing in FIG. 16B, the augment visual content is outputted 1632 tothe viewing position with the head mounted display. For example, for ahead mounted display utilizing one or more LED screens, the augmentvisual content—including physical-augment shadows therein—may bedisplayed as an arrangement of pixels forming a visible image.

Now with reference to FIG. 17, although previously herein variousembodiments have been described mainly with regard to examples in theform of methods, embodiments also may take the form of an apparatus.FIG. 17 shows an example apparatus 1770.

The apparatus 1770 includes a processor 1772, the processor 1772 beingadapted to execute executable instructions. Embodiments are not limitedwith regard to the processor 1772. A range of general-purpose,special-purpose, and embedded systems may be suitable for use as aprocessor 1772 for various embodiments. Moreover, it may be equallysuitable for the processor 1772 to consist of two or more physical orlogical processor components, or to be a “virtual” processor. Otherarrangements also may be equally suitable.

A see-through display 1774 is in communication with the processor 1772.The see-through display may be optically see-through (so as to passsensory information such as light from the physical environmenttherethrough), virtually see-through (so as to actively display sensoryinformation regarding the physical environment therefrom), or some otherarrangement. (As noted previously, embodiments are not limited only tovisual sensory content, thus audio “displays” and other output elementsmay be equally suitable. For purposes of clarity the term “see-throughdisplay” is used in certain places herein.)

A variety of see-through displays (and/or other perceive-throughoutputs) may be suitable, and embodiments are not limited with regardthereto. Suitable visual see-through-displays may include, but are notlimited to, LED, OLED, LCD, and laser projection displays. Otherarrangements, including but not limited to audio outputs (e.g.hear-through speakers), may be equally suitable.

The example apparatus 1770 in FIG. 17 also includes several dataentities 1772A, 1772B, 1772C, 1772D, 1772F, and 1772G, disposed on theprocessor 1772. The data entities 1772A, 1772B, 1772C, 1772D, 1772F, and1772G may include executable instructions instantiated on the processor,non-executable data, some combination thereof, etc. In the example shownthe data entities 1772A, 1772B, 1772C, 1772D, 1772F, and 1772G aredescribed as including executable instructions, but other arrangementsmay be equally suitable.

With regard individually to the data entities 1772A, 1772B, 1772C,1772D, 1772F, and 1772G in FIG. 17, the reference position establisher1772A is adapted to establish a reference position. The physical datamodel establisher 1772B is adapted to establish a physical data modelrepresenting at least a portion of at least one physical entity (andtypically though not necessarily a physical environment with multiplephysical entities therein). The notional data model establisher 1772C isadapted to establish a physical data model representing at least aportion of at least one notional entity (and typically though notnecessarily a notional environment with multiple notional entitiestherein).

The occlusion determiner 1772D is adapted to determine an occlusion ofat least a portion of the notional data model by at least a portion ofthe physical data model (e.g. based on whether the distance along adirection from the reference position to the physical data model is lessthan the distance along a similar direction from the reference positionto the notional data model). The occlusion applier 1772F is adaptedapply an occlusion (as determined by the occlusion determiner 1772D) tothe notional data model for purposes of output. The outputter is adaptedto deliver sensory content representing at least a portion of thenotional data model with the occlusion applied thereto to thesee-through display 1774.

For certain embodiments, including but not limited to embodimentsutilizing a virtual see-through display 1774, the outputter 1772G alsomay be adapted to deliver sensory content representing at least aportion of the physical data model.

Reference positions, physical data models, notional data models,occlusion determination, occlusion application, and output of sensorycontent with occlusion applied thereto have been previously describedherein.

Now with regard to FIG. 18, another example apparatus 1870 according isshown therein.

The apparatus 1870 includes a processor 1872, the processor 1872 beingadapted to execute executable instructions. A see-through display 1874is in communication with the processor 1872. In addition, the exampleapparatus in FIG. 18 includes a sensor 1876 in communication with theprocessor, and a data store 1878 in communication with the processor.

As has been previously noted, input from a sensor 1876 including but notcamera images such as color images and depth images may be utilized byvarious embodiments. For example, images of the physical environment mayhelp establish a physical data model, images of a person wearing a headmounted display (e.g. from a rearward facing camera aimed at thewearer's eyes) may help establish a viewing position or other referenceposition, etc. Information in a data store 1878 also may be utilized byvarious embodiments. For example, information regarding a notionalenvironment may be stored on a data store 1878, as may rules and/or datafor determining occlusion, etc. In addition, executable instructionsthat are to be instantiated on the processor (including but not limitedto data entities 1872A, 1872B, 1872C, 1872D, 1872F, and 1872G shown inFIG. 18) may be stored on a data store 1878.

Embodiments are not limited with regard to the sensor 1876 and/or thedata store 1878, nor are such elements necessarily required for allembodiments. Where a sensor 1876 is present, suitable sensors 1876 mayinclude but are not limited to monochrome cameras, color cameras,infrared cameras, depth cameras, and stereo cameras. Where a data store1878 is present, suitable data stores 1878 may include but are notlimited to hard drives and solid state drives.

In addition, although the example of FIG. 18 shows only a processor1872, see-through display 1874, sensor 1876, and data store 1878,embodiments are not limited only to such elements, and other hardwareand non-hardware elements (including but not limited to communicators,physical frameworks or bodies, etc.) also may be present in certainembodiments.

The example apparatus 1870 in FIG. 18 also includes several dataentities 1872A, 1872B, 1872C, 1872D, 1872F, and 1872G, disposed on theprocessor 1872. The reference position establisher 1872A is adapted toestablish a reference position. The physical data model establisher1872B is adapted to establish a physical data model representing atleast a portion of at least one physical entity. The notional data modelestablisher 1872C is adapted to establish a notional data modelrepresenting at least a portion of at least one notional entity. Theocclusion determiner 1872D is adapted to determine an occlusion of atleast a portion of the notional data model by at least a portion of thephysical data model. The occlusion applier 1872F is adapted to apply anocclusion (as determined by the occlusion determiner 1872D) to thenotional data model for purposes of output. The outputter 1872G isadapted to deliver sensory representing at least a portion of thenotional data model with the occlusion applied thereto to thesee-through display 1874 (and for certain embodiments also being adaptedto deliver sensory content representing at least a portion of thephysical data model).

Now with reference to FIG. 19, as has been noted occlusion may beconsidered to be an example or subset of sensory properties moregenerally that may be integrated across a physical and notional datamodel. FIG. 19 presents another example apparatus 1970, referring tosensory properties more generally.

The apparatus 1970 includes a processor 1972, the processor 1972 beingadapted to execute executable instructions. A see-through display 1974is in communication with the processor 1972. The example apparatus 1970in FIG. 19 also includes several data entities 1972A through 1972Gdisposed on the processor 1972. The data entities 1972A through 1972Gmay include executable instructions instantiated on the processor,non-executable data, some combination thereof, etc.

With regard individually to the data entities 1972A through 1972G, thereference position establisher 1972A is adapted to establish a referenceposition. The physical data model establisher 1972B is adapted toestablish a physical data model representing at least a portion of atleast one physical entity. The notional data model establisher 1772C isadapted to establish a physical data model representing at least aportion of at least one notional entity. The first sensory propertydeterminer 1972D is adapted to determine a first sensory property in oneof the physical data model and the notional data model. The secondsensory property determiner 1972E is adapted to determine a secondsensory property that at least substantially corresponds to the firstsensory property, as applicable to the notional data model. The secondsensory property applier 1972F is adapted apply the second sensoryproperty to the notional data model for purposes of output. Theoutputter 1972G is adapted to deliver sensory content representing atleast a portion of the notional data model with the occlusion appliedthereto to the see-through display 1974.

For certain embodiments, including but not limited to embodimentsutilizing a virtual see-through display 1974, the outputter 1972G alsomay be adapted to deliver sensory content representing at least aportion of the physical data model.

Now with regard to FIG. 20, another example apparatus 2070 is presentedtherein. The arrangement in FIG. 20 is at least somewhat similar to thatin FIG. 19, but is specific to the visual sense, and more so tointegrating shadows from physical light sources from a physical datamodel to a notional data model.

The apparatus 2070 includes a processor 2072, the processor 2072 beingadapted to execute executable instructions. A see-through display 2074is in communication with the processor 2072. In addition, the exampleapparatus in FIG. 18 includes a sensor 2076 in communication with theprocessor, and a data store 2078 in communication with the processor.

The example apparatus 2070 in FIG. 20 also includes several dataentities 2072A through 2072G, disposed on the processor 2072. Theviewing position establisher 2072A is adapted to establish a viewingposition. The physical data model establisher 2072B is adapted toestablish a physical data model representing at least a portion of avisual appearance of at least one physical entity. The notional datamodel establisher 2072C is adapted to establish a notional data modelrepresenting at least a portion of at least one notional entity. Thephysical-physical shadow establisher 2072D is adapted to establishshadows of physical entities as cast from physical light sources. Thephysical-notional shadow determiner 2072E is adapted to determineshadows for notional entities as cast from physical light sources, atleast substantially corresponding with the physical-physical shadows.The physical-notional shadow applier 2072F is adapted to apply thephysical-notional shadows to the notional data model for purposes ofoutput. The physical-notional shadow outputter 2072G is adapted todeliver to the see-through display 2074 sensory content representing atleast a portion of the notional data model with the physical-notionalshadows applied thereto (and for certain embodiments also being adaptedto deliver sensory content representing at least a portion of thephysical data model).

Now with reference to FIG. 21, embodiments may take many forms, and/ormay be incorporated into many devices. One example of an apparatus 2170is shown in FIG. 21, in perspective view.

As shown, the apparatus 2170 includes a processor 2172. Although notvisible in perspective view, the processor 2172 may have disposedthereon one or more data entities, such as a reference positionestablisher, physical data model establisher, notional data modelestablisher, first sensory property establisher, second sensory propertydeterminer, sensory property applier, outputter, etc., for example ascomparable to those shown and described previously with regard to FIG.19.

In addition, the example apparatus 2170 includes a body 2180, the body2180 being in the form of a pair of glasses so as to be wearable e.g. asa head mounted display. The processor 2170 is disposed on the body 2180.Also disposed on the body 180 are first and second see-through displays2174A and 2174B, and first and second sensors 2176A and 2176B. In theexample as shown, the see-through displays 2174A and 2174B are disposedon the body 2180 such that when the body 2180 is worn the displays 2174Aand 2174B would be arranged in front of, facing, and proximate the eyesof a wearer, for example so as to display stereo visual information tothat wearer. Similarly, the sensors 2176A and 2176B are shown disposedon the body 2180 such that when the body 2180 is worn the sensors 2176Aand 2176B would be arranged in left and right positions proximate andfacing substantially outward from the wearer's eyes, for example so asto capture stereo visual information at least substantially comparableto that received by the wearer's eyes.

However, the arrangement shown in FIG. 21 is an example only, and notall embodiments necessarily will have all elements shown therein or inthe configuration shown therein.

FIG. 22 is a block diagram of an apparatus that may perform variousoperations, and store various information generated and/or used by suchoperations, according to an embodiment of the disclosed technique. Theapparatus may represent any computer or processing system describedherein. The processing system 2290 is a hardware device on which any ofthe other entities, components, or services depicted in the examples ofFIG. 1 through FIG. 21 and FIG. 23 through FIG. 35 (and any othercomponents described in this specification) may be implemented. Theprocessing system 2290 includes one or more processors 2291 and memory2292 coupled to an interconnect 2293. The interconnect 2293 is shown inFIG. 22 as an abstraction that represents any one or more separatephysical buses, point to point connections, or both connected byappropriate bridges, adapters, or controllers. The interconnect 2293,therefore, may include, for example, a system bus, a PeripheralComponent Interconnect (PCI) bus or PCI-Express bus, a HyperTransport orindustry standard architecture (ISA) bus, a small computer systeminterface (SCSI) bus, a universal serial bus (USB), IIC (I2C) bus, or anInstitute of Electrical and Electronics Engineers (IEEE) standard 1394bus, also called “Firewire”.

The processor(s) 2291 is/are the central processing unit of theprocessing system 2290 and, thus, control the overall operation of theprocessing system 2290. In certain embodiments, the processor(s) 2291accomplish this by executing software or firmware stored in memory 2292.The processor(s) 2291 may be, or may include, one or more programmablegeneral-purpose or special-purpose microprocessors, digital signalprocessors (DSPs), programmable controllers, application specificintegrated circuits (ASICs), programmable logic devices (PLDs), trustedplatform modules (TPMs), or the like, or a combination of such devices.

The memory 2292 is or includes the main memory of the processing system2290. The memory 2292 represents any form of random access memory (RAM),read-only memory (ROM), flash memory, or the like, or a combination ofsuch devices. In use, the memory 2292 may contain a code. In oneembodiment, the code includes a general programming module configured torecognize the general-purpose program received via the computer businterface, and prepare the general-purpose program for execution at theprocessor. In another embodiment, the general programming module may beimplemented using hardware circuitry such as ASICs, PLDs, orfield-programmable gate arrays (FPGAs).

The network adapter 2294, a storage device(s) 2295, and I/O device(s)2296, are also connected to the processor(s) 2291 through theinterconnect 2293 The network adapter 2294 provides the processingsystem 2290 with the ability to communicate with remote devices over anetwork and may be, for example, an Ethernet adapter or Fibre Channeladapter. The network adapter 2294 may also provide the processing system2290 with the ability to communicate with other computers within thecluster. In some embodiments, the processing system 2290 may use morethan one network adapter to deal with the communications within andoutside of the cluster separately.

The I/O device(s) 2296 can include, for example, a keyboard, a mouse orother pointing device, disk drives, printers, a scanner, and other inputand/or output devices, including a display device. The I/O device(s)2296 also may include, for example, cameras and/or other imagers adaptedto accept visual input including but not limited to postures and/orgestures. The display device may include, for example, a cathode raytube (CRT), liquid crystal display (LCD), or some other applicable knownor convenient display device. The display device may take various forms,including but not limited to stereo displays suited for use in near-eyeapplications such as head mounted displays or other wearable devices.

The code stored in memory 2292 may be implemented as software and/orfirmware to program the processor(s) 2291 to carry out actions describedherein. In certain embodiments, such software or firmware may beinitially provided to the processing system 2290 by downloading from aremote system through the processing system 2290 (e.g., via networkadapter 2294).

The techniques herein may be implemented by, for example, programmablecircuitry (e.g. one or more microprocessors) programmed with softwareand/or firmware, or entirely in special-purpose hardwired(non-programmable) circuitry, or in a combination of such forms.Special-purpose hardwired circuitry may be in the form of, for example,one or more AISCs, PLDs, FPGAs, etc.

Software or firmware for use in implementing the techniques introducedhere may be stored on a machine-readable storage medium and may beexecuted by one or more general-purpose or special-purpose programmablemicroprocessors. A “machine-readable storage medium”, as the term isused herein, includes any mechanism that can store information in a formaccessible by a machine.

A machine can also be a server computer, a client computer, a personalcomputer (PC), a tablet PC, a laptop computer, a set-top box (STB), apersonal digital assistant (PDA), a cellular telephone, an iPhone, aBlackberry, a processor, a telephone, a web appliance, a network router,switch, or bridge, or any machine capable of executing a set ofinstructions (sequential or otherwise) that specify actions to be takenby that machine.

A machine-accessible storage medium or a storage device(s) 2295includes, for example, recordable/non-recordable media (e.g., ROM; RAM;magnetic disk storage media; optical storage media; flash memorydevices; etc.), etc., or any combination thereof. The storage mediumtypically may be non-transitory or include a non-transitory device. Inthis context, a non-transitory storage medium may include a device thatis tangible, meaning that the device has a concrete physical form,although the device may change its physical state. Thus, for example,non-transitory refers to a device remaining tangible despite this changein state.

The term “logic”, as used herein, may include, for example, programmablecircuitry programmed with specific software and/or firmware,special-purpose hardwired circuitry, or a combination thereof.

Now with reference to FIG. 23 through FIG. 31B collectively, aparticular instance of data integration is illustrated and described asan example application of certain embodiments. The instance describedwith regard to FIG. 23 through FIG. 31B relates to identifying shadowsfrom physical objects cast from physical light sources, and applyingcorresponding notional shadows onto notional objects. The example inFIG. 23 through FIG. 31B is described in terms of a hand as a physicalobject, a virtual/augment data tablet as a notional object, and a flatsurface such as a table or counter upon which the shadow of the hand maybe cast from a physical lamp as a light source. However, these areexamples only, provided for clarity, and are not limiting.

Referring specifically to FIG. 23, a surface 2342 such as a table, acounter, a white board, etc. is shown. A feature 2343 also is visible onthe surface 2342; the feature 2343 is in the shape of a hand withfingers extended and partially spread. In some instances, such a feature2343 may be a shadow, for example a shadow cast by a human hand.However, a feature 2343 in the shape of a hand is not necessarily ashadow; a hand-shape also may be drawn onto a surface, sketched onto asheet of paper placed on a table, applied as an adhesive label (e.g. toindicate “place your hand here”, etc.), and so forth. More generallyspeaking, a given visible feature 2343 may be a shadow, but may besomething else instead, such as some form of visible marking.

A distinction between shadows and markings may be significant, asexplained now. With reference to FIG. 24, a similar surface 2442 to thatshown in FIG. 23 is visible. A virtual/augment tablet 2410 (an exampleof a notional entity) is also visible, some distance above the surface2442. No hand-shaped feature is shown in FIG. 24, but examples of suchare presented in FIG. 25 and FIG. 26, for different types of hand-shapedfeatures.

In FIG. 25, a surface 2542 and virtual/augment tablet 2510 are visible.In addition, a hand-shaped feature 2543 is shown disposed on the surface2542. The virtual/augment tablet 2510 occludes the hand-shaped feature2543. Such behavior is as may be expected of a physical tabletinteracting with a hand-shaped marking. That is, if the hand-shapedfeature 2543 is a marking, such as a drawing, a label, a discoloration,some other surface feature, etc., then if a real-world tablet were heldabove the hand-shaped feature 2543, that real-world tablet would coversome or all of the hand-shaped feature 2543.

By contrast, in FIG. 26 a surface 2642 and a virtual/augment tablet 2610again are visible. However, a hand-shaped shadow 2648 is shown disposednot on the surface 2642 but on the virtual/augment tablet 2610. Suchbehavior is as may be expected of a physical tablet interacting with aprojected shadow. That is, if a hand-shaped feature on a surface 2642 isa shadow projected onto that surface 2642, and a physical tablet wereplaced on or above the surface 2642, then some or all of a hand-shapedshadow 2648 would appear on the tablet.

FIG. 25 and FIG. 26 thus illustrate two different phenomena that mayresult, depending on whether a feature on a surface is a marking on thesurface (as in FIG. 25) or is a shadow projected onto the surface (as inFIG. 26). Typically for physical environments, surface markings arecovered by (e.g. occluded by) overlying objects, while projected shadowscover overlying objects.

In order to integrate notional entities with physical entities toprovide an appearance that both notional and physical entities are“present”, following the same rules of reality, etc., it may be usefulto produce similar effects for a virtual/augment tablet as may beexpected for a physical tablet. That is, for a virtual/augment tabletdisposed above a physical surface, a projected physical shadow shouldcover that virtual/augment tablet, while a marking on the surface shouldbe covered by the virtual/augment tablet.

One approach for enabling such integration with certain embodiments maybe understood from FIG. 27 and FIG. 28.

In FIG. 27, a view of a physical environment is shown that is at leastsomewhat similar to the arrangements in FIG. 23 through FIG. 26, exceptthat the view shown reflects only volumetric or “solid” objects. Asurface 2742 is shown, such as a table or counter. A human hand (e.g. aphysical entity) 2709 is shown disposed some distance above the surface2742. A light source 2744 in the form of a lamp is shown above the hand2709, such that light from the lamp 2744 may project a shadow of thehand 2709 onto the surface 2742. However, no shadow is visible in FIG.27; a shadow, not being volumetric, typically is not visible involumetric or solids data.

Data such as that illustrated in FIG. 27 may be obtained for examplefrom a depth camera, which may be adapted to determine the depths tovarious objects, surfaces, etc., thus potentially determining the size,position, arrangement, shape, etc. of such entities but typically notdetermining surface information. Thus, a depth camera image of aphysical environment may reveal the surface 2742, the hand 2709, and thelamp 2744 as shown in FIG. 27, but may not reveal whether a hand-shapedfeature is present on the surface 2742, regardless of whether such afeature is a shadow, a surface marking, etc. Similarly, while such adepth camera image may reveal the physical form of a lamp 2744, thatdepth camera image may not reveal whether illumination is beingproduced, the direction, color, or spread of light, whether light isstriking the hand 2709 or other entities, etc.

Again in FIG. 28, a view of a physical environment is shown that is atleast somewhat similar to the arrangement in FIG. 27, except that theview shown reflects only surface data. That is, color, brightness, etc.may be present, but depth and volumetric data may not be present (atleast not directly). Thus in FIG. 28, contours are not visible; inpractice contours might be visible as variations in color, shade, etc.,but for simplicity coloring and shading are not shown in FIG. 28. Asurface 2842 is shown, with a human hand 2809 disposed some distanceabove the surface 2842 and a light source 2844 above the hand 2809. Afeature 2843 in the shape of a hand is also shown on the surface 2842;such a feature 2843 may be visible as being darker than the remainder ofthe surface 2842, being a different color than the remainder of thesurface 2842, etc. The feature 2843 may be a shadow, or the feature 2843may be surface mark; either may be possible given the arrangement inFIG. 28.

However, it is noted that surface data such as that in FIG. 28 mayinclude additional information that may help to determine whether thefeature 2843 is a surface mark or a shadow (or some combination of both,or even some other entity). For example, although color and shading areomitted from FIG. 28 as noted, in practice a surfaces image such as thatin FIG. 28 may indicate through color, shading, or other featureswhether the lamp 2844 is lit, how bright that lamp 2844 is (if lit), thecolor, direction, focus, etc. of light from the lamp 2844, etc. This mayinclude color or brightness data at or near the lamp 2844 itself, butalso may include color or brightness data on the hand 2809, on thesurface 2842, or on other aspects of the environment (if present). Forexample, if the hand 2809 is brighter on top than on the bottom (or onside but not the other, etc.), this may suggest the presence of anactive light source above the hand 2809 (or to one side, etc.).Similarly, a bright circle or “halo” on the surface 2842 as of impinginglight, or a bright reflection of a light on the surface 2842 (if thesurface is reflective), may suggest the presence, direction, etc. of anactive light source.

Although considering volumetric data alone or surface data alone may notreveal whether a given feature is a shadow, consideration of bothvolumetric and surfaces data may be sufficient. For example, if lightsources and objects that may cast shadows from those light sources maybe identified, then a determination may be made as to whether a shadowcould and/or should be present on some surface. This may in turn betaken as evidence that a given feature is or is not a shadow; if aphysical object of suitable size, shape, and position is disposedrelative to a surface and a light source as to produce a shadow, and thefeature in question matches the shadow that would be so produced, it maybe concluded that the feature is indeed a shadow. Conversely, if aphysical environment is such that a feature does not appear tocorrespond with any shadows that should be present, it may be concludedthat the feature is not a shadow.

Other factors also may be considered. For example, shadows typically maybe darker than their surroundings. That is, a shadow of a hand on a graycounter top may be a darker shade of gray than the surrounding area, buttypically will not be a brighter shade of gray. As another example, fortypical “white” lighting in common indoor use shadows generally are notcolored differently from their surroundings. That is, a green marking ona white surface may, in at least some circumstances, be concluded to besomething other than a shadow (a marking, a reflection, a “hot spot”from a light source such as a laser pointer, etc.).

With reference to FIG. 29A and FIG. 29B, it is noted that it is notnecessary that light sources be explicitly present in an environment oran image thereof in order to be considered. That is, in the arrangementsin FIG. 27 and FIG. 28 the lamp 2744/2844 is visibly present. However,in FIG. 29A a similar view is shown, with a surface 2942A, a hand-shapedfeature 2943A, and a hand 2909A. No light source is explicitly present;if only entities that are explicitly present in FIG. 29A are considered,then it may be concluded that the feature 2943A is not a shadow becauseno suitable light source exists to cast such a shadow.

However, as noted previously light may be determined throughconsideration of color, brightness, etc., and thus a light source may bedetermined to be implicitly present even if not explicitly visible. Forexample, if the hand 2909A is brighter on top than on the bottom, then alight source may be concluded to be present above the hand.

Referring now to FIG. 29B, an arrangement similar to that in FIG. 29Abut pulled back and from a slightly different perspective is shown.Again the surface 2942B, hand-shaped feature 2943B, and hand 2909B areshown, but as may be seen in FIG. 29B a light source 2944B also ispresent. Even if such a light source is not visibly present in a givenview of or data set for an environment (as in FIG. 29A), the lightsource 2944B may be determined to be implicitly present. As noted alight source 2944B may be inferred to be present because of color orbrightness aspects of a surfaces image. Reflections of a light sourceand/or other factors also may be considered.

In addition, if a light source 2944B was previously determined to bepresent, for example having been visible in some previous view of theenvironment, then that light source 2944B may be considered as beingimplicitly present even if not in view at a particular moment. Likewise,if one or more shadows are known to be present, were previouslyidentified, etc., then the presence, location, etc. of a light source2944B may be inferred from those other shadows.

Such an arrangement may be referred to as “persistence”. That is, if alight source has been imaged, inferred, etc. at some previous time, theexistence of the light source may be noted and/or retained for futureconsideration. As a more concrete example, if a lamp were identified inone image based on color or brightness, then features such as theposition, orientation, brightness, color, etc. of that lamp may bestored as data within a processor, or recorded in a data store. Thatdata describing the lamp may be considered in the future, even if themost recent images or other data do not explicitly show the presence ofthat particular lamp.

For instance, if a user wearing a head mounted display with a colorcamera thereon were to turn away from such a lamp, the lamp may cease tobe present within the color camera's field of view, but the lamp maystill be considered when addressing shadowing, etc. Even if the colorcamera and/or the user cannot see the lamp at a given time, that lampstill may affect shadows that the camera and/or the user can see. Thus,some or all information regarding some or all light sources may beretained as persistent information, usable for later consideration ofshadows (or other phenomena).

No specific time or circumstance limit will necessarily apply topersistent light source data. In certain embodiments, it may be usefulto retain light source data as persistent for several minutes or forsome other duration, for example on the expectation that a user may notrapidly move out of the effective range of a light source. In otherembodiments, it may be useful to retain or discard persistent lightsource data based on the position of a device, sensor, user, etc., asdetermined for example through GPS or other avenues. In still otherembodiments, light source data may be persistent indefinitely, forexample being stored for individual locations such as a user's desk attheir place of work, various rooms in their home, their automobile, etc.In such instance, light sources may be anticipated to remain at leastsomewhat consistent over time, e.g. the location of built-in lightfixtures may be essentially permanent, while even floor lamps andsimilar may remain at or near a single position for long periods. Suchpersistent light source data may be useful for example in enhancingintegration of data sets (e.g. because the positions and otherproperties of light sources in a given location frequently visited bythe user may be well-defined with high accuracy and/or precision), mayreduce computational requirements (if some or all light sources arealready characterized, then determining light sources “on the fly” maybe avoided), etc.

In more colloquial terms, just because a light source is not currentlyvisible does not necessarily mean that no light sources are present.Embodiments may consider light sources that are not directly visible,but for which evidence exists, or which are otherwise considered likelyto be present (e.g. overhead lighting may be assumed in interior officespaces, the sun may be assumed present and in a particular positionbased on GPS data and local time, etc.).

In addition, physical entities whether implicit or explicit also may beconsidered as persistent, in similar fashion to persistent light sourcesas noted above. More broadly, solids data, surfaces data, physicalmodels, notional models, etc. may, for certain embodiments, be heldpersistent in whole or in part.

As an aside, it is noted that it may not be required for all embodimentsto necessarily determine certain features, for example light sources.Given some other approach for determining whether an unknown feature isa shadow, a surface image, an imaging error, etc., it may not benecessary to determine whether light sources are present, where thoselight sources are, etc. As a more concrete example, it may be suitablein certain embodiments to distinguish shadows from other features basedon darkness, color, sharpness/blurriness of edges, etc., or throughalgorithms that determine (either based on concrete factors or throughabstract analysis) whether a feature likely is or likely is not ashadow. Other approaches also may be equally suitable.

Thus, although particular example descriptions are provided to indicateapproaches for identifying shadows and thus enabling integration ofshadows among physical and notional data sets, the approaches describedherein should not be considered as limiting, and other arrangements maybe equally suitable.

With regard to FIG. 30A and FIG. 30B, similarly objects casting shadowsmay be implicit rather than explicit. In FIG. 30A a surface 3042A and ahand-shaped feature 3043A are shown. No light source is explicitlypresent, nor a hand; if only entities that are explicitly present inFIG. 30A are considered, then it may be concluded that the feature 3043Ais not a shadow because no suitable light source exists to cast such ashadow and no suitable object exists to produce such a shadow.

However, as with light sources physical objects may be determined asbeing implicitly present, even if not explicitly visible. For example,if the environment is being viewed from a head-mounted display, then itmay be anticipated as likely that a user's hands may be present nearby,and (e.g. based on anatomical limitations) that the user's hands may belimited to a range of possible positions, orientations, etc.

With reference to FIG. 30B, an arrangement similar to that in FIG. 30Abut pulled back and from a slightly different perspective is shown.Again the surface 3042B and hand-shaped feature 3043B are shown, but asmay be seen in FIG. 30B a light source 3044B and a hand 3009B also arepresent. As with light sources, hands and/or other physical objects maybe determined to be implicitly present, even if not necessarilyimmediately detected. The presence of a user, body parts thereof, anembodiment of apparatus itself, other objects, etc. may be assumed toexist, for example. Objects previously detected may be considered tostill be present, objects may be predicted as being likely to bepresent, etc.

Now with reference to FIG. 31A and FIG. 31B, as noted it may be usefulto determine whether a given feature is a shadow or a surface feature,for example so as to adjust the appearance of notional entities tocorrespond with behavior expected for physical entities. That is, avirtual/augment shadow may be added to a virtual/augment tablet if aphysical shadow is concluded to be present, where a virtual/augmenttablet would instead occlude a physical surface marking.

However, embodiments are not necessarily limited only to replicatingdetected phenomena.

In FIG. 31A, a surface 3142A is shown with a virtual/augment tablet3110A disposed above, and a hand 3109A disposed above thevirtual/augment tablet 3110A. A light source 3144A is in turn disposedabove the hand 3109A. A hand-shaped virtual/augment shadow 3159A isshown projected onto the virtual/augment tablet 3110A, so as tocorrespond with what may be expected for a physical tablet; that is, thevirtual/augment tablet 3110A is made to look “more real” by exhibiting ashadow of the hand 3109A thereon.

The shadow 3159A is a notional entity, but is made to appear as thoughcast by the physical light source 3144A from the physical hand 3109Aonto the virtual/augment tablet 3110A. To recall terminology fromprevious examples, the shadow 3159A may be referred to as a“physical-physical-notional” entity: physical light, physical object,notional surface. More colloquially, the shadow 3159 is a notionalshadow that mimics a shadow as may be cast by a physical light from aphysical object, falling on a notional surface.

However, as may be seen a tablet-shaped virtual/augment shadow 3148Aalso is present. This corresponds to a shadow as may be cast by thephysical light 3144A from the virtual/augment tablet 3110A, falling onthe physical surface 3142A.

Considering only physical factors, a hand-shaped shadow cast by thephysical light 3144A from the physical hand 3109A onto the physicalsurface 3142A may be expected. However, if a physical tablet is present,then that hand-shaped shadow may be covered by a shadow in the shape ofthat physical tablet. Thus the virtual/augment shadow 3148A representsin some sense a shadow correction: although the expected shadow 3158A onthe virtual/augment tablet 3110A is already considered, a further shadow3148A may be added to further integrate the virtual/augment tablet withthe physical environment.

To again recall previous terminology, the shadow correction 3148A may bereferred to as a “physical-notional-physical” entity: physical light,notional object, physical surface.

With reference to FIG. 31B, shadow correction is not limited only topositive effects such as adding further shadows. For certainembodiments, it may be useful to counter-act existing shadows, forexample by brightening or “bleaching” a shadow with a notional object(e.g. a brightening “mask” in the shape of the shadow in question).

In FIG. 31B, a surface 3142B, virtual/augment tablet 3110B, hand 3109B,light source 3144B, and hand-shaped virtual/augment shadow 3159B areshown. However, no tablet-shaped virtual/augment shadow 3148A is shown.Instead, a shadow correction in the form of a “counter shadow” 3149B ispresent. The counter shadow 3149B is a virtual/augment entity thatbrightens or “bleaches” the shadow that is cast by the physical hand3109B; as a result the shadow of the physical hand is made lessnoticeable, possibly even invisible. The shadow correction 3149B thusmay be considered a sort of “negative shadow”.

To again recall previous terminology, the shadow correction 3149B may bereferred to as a “physical-notional-physical” entity: physical light,notional object, physical surface, though in some sense the shadowcorrection 3149B is an inverse physical-notional-physical entity,removing or diminishing a feature (a visible shadow) rather than addingone. However, such negative shadows or other inverse entities are notprohibited.

It is noted that the term “shadow correction” is descriptive, but shouldnot be considered limiting. A “shadow correction” is a correctioninsofar as further altering the appearance of a mixed physical andvirtual/augment environment to either more closely approximate afully-physical environment, or at least avoid incongruity thereof. Thatis, adding a virtual/augment shadow of the virtual/augment tablet on thephysical surface as cast by the physical light may be considered to moreclosely approximate a fully integrated environment: a “real” tabletwould cast a shadow, thus the virtual/augment tablet is made to appearto cast a shadow.

However, as noted full integration is not necessarily required. In“bleaching” a hand shadow with a virtual/augment shadow correction, theappearance of a hand shadow “where it shouldn't be” may be avoided.Though the lack of a shadow of a virtual/augment tablet may not beentirely realistic, the presence of a hand shadow may be moredistracting and/or otherwise incongruous to a viewer than the absence ofshadow for the tablet.

Now with reference to FIG. 32A and FIG. 32B, therein an example methodfor integrating physical and notional content is shown. The method inFIG. 32A and FIG. 32B specifically addresses identifying shadows in aphysical environment and integrating suitable corresponding notionalshadows, and in particular to carrying out such tasks with a headmounted display having a processor therein. The arrangement shown inFIG. 32A and FIG. 32B bears some overall similarity to the method shownin FIG. 15, though this should not be considered limiting (nor is thearrangement in FIG. 15 itself limiting), but rather an example only. Inparticular, embodiments are not limited only to integrating shadow, norto the approach shown when integrating shadows, nor to the use of headmounted displays.

In FIG. 32A, a viewing position is established 3220 for a head mounteddisplay. The reference position serves as a baseline for integratingcontent. In practice such a viewing position may include multiplerelated positions, for example the position of the display with respectto the viewer's eyes, the position of the display with respect to aphysical environment, a nominal viewing position within avirtual/augmented reality environment, some combination thereof, etc.Likewise, for a head mounted display having stereo output capability,there may be two (or more) viewing positions, e.g. left and right stereoviewing positions. However for simplicity the viewing position isreferred to herein as singular.

A depth camera image of a physical environment is received 3222A in theprocessor of the head mounted display. Typically though not necessarilythis may be obtained through the use of a depth camera such as a digitaltime-of-flight camera disposed on the head mounted display, capturing aspatial map of some part of the physical environment. The depth imageincludes position information, and thus volumetric information regardingthe environment, but typically does not include surface information suchas color, brightness, etc.

Use of a depth camera is an example only. Other approaches may includebut are not limited to use of a stereo camera, a plenoptic camera, adiscrete time-of-flight sensor, an ultrasonic sensor, and a millimeterwave sensor. Other arrangements may be equally suitable.

Solids data is determined from the depth image 3222B in the processor.Solids data constitutes a model of the physical environment, with regardto size, shape, position, etc. Solids data also may be considered to bevolumetric data. Alternately, solids data may be considered to be anaspect off a more comprehensive physical model, for example a physicalmodel that combines both solids data and surfaces data (describedbelow). Step 3222B may be, but is not required to be, inherent in thereceipt of the depth image 3222A. That is, the depth image itself may bereceived in such form as to serve without further processing as solidsdata. However, even so further modification may be made by theprocessor, for example removing sensor noise from the depth image,combining the received depth image with additional information, etc., soas to produce (or expand) the solids data.

A color camera image of the physical environment is received 3222C inthe processor of the head mounted display. Typically though notnecessarily this may be obtained through the use of a color camera suchas a digital RGB camera disposed on the head mounted display, capturingan image of some part of the physical environment. (Typically the colorcamera image will have a field of view similar or identical to the fieldof view of the depth camera image, though this is not required.) Thecolor camera image includes surface information, such as color,brightness, etc., but may not include depth or other direct volumetricor 3D positional information.

Use of a color image camera is an example only. Other approaches mayinclude but are not limited to use of a grayscale camera, ablack-and-white camera, a digital camera, and an infrared camera. Otherarrangements may be equally suitable.

Surfaces data is determined from the color camera image 3222D in theprocessor. Surfaces data constitutes a model of the physicalenvironment, with regard to brightness, color, etc. Alternately,surfaces data may be considered to be an aspect off a more comprehensivephysical model, for example a physical model that combines both solidsdata and surfaces data. Step 3222D may be, but is not required to be,inherent in the receipt of the color camera image 3222C. That is, thecolor camera image itself may be received in such form as to servewithout further processing as surfaces data. However, even so furthermodification may be made by the processor.

Collectively, steps 3222A through 3222D may be considered at leastsomewhat analogous to step 1522 in FIG. 15, though this similarity maynot be exact and should not be considered limiting.

Continuing in FIG. 32A, an augment visual model, i.e. informationregarding the existence, appearance, position, etc. of augmented realityentities, is generated 3224 in the processor.

Potential shadows are identified 3225A in the surfaces data based oncolor in the processor. The term “color” as applied here (and likewiseto step 3225B) should be understood to include not only shade but alsobrightness/darkness, so that a feature in the surfaces data may beidentified as a potential shadow based on whether that feature is darkerthan whatever surrounds that feature, etc. Identification of potentialshadows may be simple, e.g. some arbitrary difference incolor/brightness, or may be more complex. For example shadowidentification may address color distributions that consider thesharpness/fuzziness of outlines, minimum and maximum differences incolor (since shadows typically are not completely black, objects thatare too dark may be ignored), smoothness or roughness of edges of colorvariations, etc.

The notion of a “potential shadow” refers to further consideration as towhether a feature is, indeed, a shadow and should be treated as such.Potential shadows are not necessarily automatically assumed to be actualshadows (though this is not excluded), but typically are candidates forevaluation.

Light sources are identified 3225B in the surfaces data based on colorin the processor. For example, a region of the surfaces data exhibitinghigh brightness, or even off-scale brightness (e.g. having saturated thecolor camera at that point), may be considered to be a light source. Aswith potential shadow identification 3225A, light source identification3225B may be simple, such as a minimum brightness threshold, or complex.For example, patterns of color/brightness may be sought that representillumination of an object, rather than necessarily seeking the lightsource proper (as noted previously, light sources may be implicit andnot explicitly visible within a set of surface data), reflections may beidentified, haloing on surfaces or apparent beams due to dust suspendedin air, etc. may be considered, and so forth.

Light sources identified in step 3225B are projected from the surfacesdata onto/into the solids data 3225C in the processor. That is, based onsignatures of color/brightness in the surfaces data that indicate thepresence of a light source, the light source is then located within thesolids data. This may be a simple geometric projection, e.g. the x-yposition in surfaces data identified as a light source may be projectedoutward through the solids data until reaching some solid feature, anddesignating that solid feature as a light source. However, againprojection 3225C is not limited only to such direct approaches, and maybe more complex. For example, light sources identified within thesurfaces data may be projected into the solids data based not only onposition but also on the shape of objects or features within the solidsdata. More colloquially, some solid object may be identified as a lightbased in whole or in part on whether that object is shaped like a knownlight. Other approaches also may be suitable.

Anticipated shadows are determined 3225D from light sources and solidsdata in the processor. That is, based on the locations, directions, etc.of light sources in 3D space (as projected in step 3225C), and thesizes, positions, etc. of intervening objects in 3D space (within thesolids data), it is determined at least approximately where, how large,and what shape of shadows should result from such light sources andintervening objects. Typically though not necessarily this may beaccomplished through ray-tracing.

Now with reference to FIG. 32B, a determination is made 3225E in theprocessor as to whether a given potential shadow matches with ananticipated shadow. That is, does a feature that was identified aspossibly being a shadow (in step 3225A) fit any of the shadows that weredetermined should exist given the known light sources and interveningobjects (determined in step 3225D)? An absolute match is not necessarilyrequired (though absolute matches are not excluded). Some degree of“misfit” may be suitable, for example to accommodate limited data,sensor noise, limited processing power, inaccuracy in solids and/orsurfaces data, etc.

If a given potential shadow does not match any anticipated shadow—if thedetermination 3225E is negative—then the method is complete, at leastinsofar as that potential shadow is concerned. The arrangement in FIG.32A and FIG. 32B may in some sense be considered a filtering approach:features are identified as possibly being shadows, and then either arefound to match with predicted shadows and treated as in fact beingshadows, or are found not to match and are ignored. However, this is anexample only, and other approaches may be suitable.

If a given potential shadow does match an anticipated shadow—if thedetermination 3225E is positive—then the method continues with step3226A.

Collectively, steps 3225A through 3225E may be considered at leastsomewhat analogous to step 1525 in FIG. 15, though this similarity maynot be exact and should not be considered limiting.

Continuing in FIG. 32B, for a given potential shadow that matches ananticipated shadow, that potential shadow is identified 3226A as anoccluding shadow in the processor. Typically though not necessarily suchidentification may be handled by “tagging” the potential shadow, theregion of the surfaces data exhibiting the potential shadow, etc. so asto indicate that the potential shadow has been verified and is to betreated differently, i.e. as a “definite” shadow and not merely apotential shadow.

The occluding shadow is (or potentially multiple occluding shadows are)then applied 3226B to the augment visual model in the processor. Thatis, where previously the potential shadow and later the occluding shadowwere part of the surfaces data reflective of the physical environment,now the occluding shadow is added to the augment visual model as avisible feature of the augmented reality environment. Again, this may behandled by “tagging”, e.g. by identifying whatever data represents theoccluding shadow as now also being part of the augment visual model.However, other approaches, including but not limited to copying theoccluding shadow into the augment visual model, also may be suitable.

Augment visual content s generated 3228 with the occluding shadow(s)applied thereto, in the processor. That is, some image, render, etc. ofthe augment visual model is produced so as to be suitable for output.

The augment visual content is outputted 3232 to the reference positionwith the head mounted display. Thus, for example a virtual/augmentshadow such as 3159A in FIG. 31A may be integrated with avirtual/augment tablet 3110A as also shown in FIG. 31A, presenting anappearance that a physical light and a physical hand are casting ashadow onto an augmented reality entity. In such manner, shadowing maybe partially or fully integrated for an augmented reality and a physicalreality, e.g. producing the virtual/augment shadow 3159A in FIG. 31A.However, this is an example only, and other arrangements may be equallysuitable.

Now with reference to FIG. 33A and FIG. 33B, a somewhat more generalizedmethod than that in FIG. 32A and FIG. 32B is shown, so as to presentexamples of certain options and/or variations, though still with regardto shadowing in somewhat similar fashion to that shown in FIG. 32A andFIG. 32B.

In FIG. 33A, a viewing position is established 3220 for a head mounteddisplay.

Solids data is established 3322A for an environment in a processor, andsurfaces data is established in the processor. In the previous exampleof FIG. 32A solids data was determined from a depth image that wasobtained of the environment. and surfaces data was determined from acolor camera image that was obtained of the environment; while suchapproaches may be suitable for certain embodiments, embodiments are notlimited only thereto. Other approaches for establishing solids data maybe equally suitable, including but not limited to determining solidsdata and/or surfaces data from other sources (e.g. stereo imagery inaddition to/instead of a depth camera image for solids data, etc.),reading solids data from a data store such as a hard drive or solidstate drive, or receiving solids data from some external source (e.g.through wired or wireless communication).

An augment virtual model is established 3324 in the processor.

Potential shadows are identified 3225A in the surfaces data with theprocessor, and light sources also are identified 3225B in the surfacesdata with the processor. In the previous example of FIG. 32A potentialshadows and light sources were identified by color; while suchapproaches may be suitable for certain embodiments, embodiments are notlimited only thereto. Other approaches for identifying potential shadowsand/or light sources data may be equally suitable, including but notlimited to determining solids data and/or surfaces data other than bycolor (e.g. from shape, through mathematical analysis, through abstractlearning algorithms, etc.), reading data regarding potential shadowsand/or light sources from a data store such as a hard drive or solidstate drive, or receiving data regarding potential shadows and/or lightsources from some external source.

Light sources are mapped 3225C from surface data onto solids data in theprocessor. In the previous example of FIG. 32A light sources wereprojected into the solids data from the surfaces data; while suchapproaches may be suitable for certain embodiments, embodiments are notlimited only thereto. Other approaches for locating in the solids datapositions for light sources that are identifiable within the surfacedata may be equally suitable, including but not limited to mappingapproaches other than projection, reading data regarding light sourcelocation from a data store such as a hard drive or solid state drive, orreceiving data regarding light source locations from some externalsource.

Still with reference to FIG. 33A anticipated shadows are determined3325D based on the light sources and solids data, and then moving on toFIG. 33B a determination is made 3325E as to whether one or morepotential shadows matches the anticipated shadows. Potential shadowsthat match an anticipated shadow are identified 3326A as occludingshadows, and occluding shadows are applied 3326B to the augment model inthe processor. Augment visual content is generated 3328 including theoccluding shadows, and is outputted 3332 to the reference position withthe head mounted display.

Now with reference to FIG. 34A and FIG. 34B, as noted previously manyvarying embodiments may be suitable. FIG. 34A and FIG. 34B show anarrangement that may be somewhat similar overall with FIG. 32A and FIG.32B, but that shows different details. Embodiments are not limited onlyto the particulars in FIG. 32A and FIG. 32B or in FIG. 34A and FIG. 34B,however, and the arrangements shown therein are examples only.

In FIG. 34A, a viewing position for a head mounted display isestablished 3420. A pair of grayscale stereo camera images are received3422A in a processor of the head mounted display, and solids data aredetermined 3422B in the processor from the stereo image pair. Surfacesdata also are determined 3442C from one or both of the grayscale stereoimages. Where in the example of FIG. 32A and FIG. 32B solids data andsurfaces data each were determined from a distinct image/input, namely adepth image for solids data and a color image for surfaces data, asshown in FIG. 34A the use of distinct or independent sources is notrequired. For certain embodiments it may be suitable to acquire bothsolids data and surfaces data from a single source, for example a pairof stereo images as shown.

An augment visual model is read 3424 into the processor from a datastore such as a hard drive or solid state drive. As noted previously,embodiments are not limited with regard to how an augment visual modelis established, nor necessarily to what model is established (e.g.visual, auditory, etc., augmented reality, virtual reality, etc.).

Potential shadows are identified 3425A in the surfaces data based ongeometry, with the processor. That is, features are identified aspossibly being shadows based on the size, shape, position, etc. of thosefeatures. Light sources are identified 3425B in the surfaces dataaccording to brightness, with the processor. Although in certainprevious examples potential shadows and light sources have beenidentified similarly (e.g. both based on color), as may be seen throughcomparing steps 3425A and 3425B it is not required that potentialshadows and light sources are identified through the same or similarapproaches, i.e. in FIG. 34A potential shadows are addressed throughgeometry while light sources or addressed based on image brightness.

Still with reference to FIG. 34A, light sources are triangulated 3425Cwithin the solids data in the processor. In the example of FIG. 34A thesolids data is established through consideration of two stereo images,light sources may be determined by brightness in one or both images, andpositions of those light sources in three dimensional space determinedthrough triangulation. As noted, embodiments are not limited todetermining light position in the solids data through projection fromthe surfaces data, nor even necessarily through consideration of thesurfaces data at all.

Anticipated shadows are determined 3425D in the processor from lightsources and solids data. Moving on to FIG. 34B a determination is made3425D as to whether one or more potential shadows matches theanticipated shadows. Potential shadows that match an anticipated shadoware identified 3426A as occluding shadows, and occluding shadows areapplied 3426B to the augment model in the processor.

A shadow correction is determined 3427 in the processor from the lightsources and the augment model. For example, to refer to FIG. 31A ashadow that may be expected to be cast the augment tablet therein may bedetermined, based on the position of the light source therein and theshape and size of the tablet. However, such an approach is an exampleonly; determining a counter-shadow for a physical hand as in FIG. 31Bmay not require consideration of an augment model but may require or atleast benefit from consideration of the solids model, since both thelight source and the object casting the shadow (in FIG. 31B, a hand) arephysical, as is the surface onto which the shadow is cast. Otherarrangements also may be suitable.

Augment visual content is generated 3428 including the occludingshadows, and is outputted 3432 to the reference position with the headmounted display.

Now with reference to FIG. 35, an example embodiment of an apparatus isshown, at least approximately corresponding with certain examples as inFIG. 32A through FIG. 34B. The apparatus 3570 includes a processor 3572,an image sensor (which may include but is not limited to a color camera)in communication with the processor 3570, a depth sensor 3577 (which mayinclude but is not limited to a depth camera) in communication with theprocessor 3570, and an optical see-through display 3574 (including butnot limited to a stereo head mounted display) in communication with theprocessor 3570.

A number of data entities 3572A through 3572N are shown disposed on theprocessor 3572, as follows. The data entities 3572A through 3572N mayinclude executable instructions instantiated on the processor,non-executable data, some combination thereof, etc., and otherarrangements may be equally suitable.

A viewing position establisher 3572A is instantiated on the processor3572, and is adapted to establish a viewing position. A solids dataestablisher 3572B is instantiated on the processor 3572, and is adaptedto establish solids data, e.g. from one or more readings received fromthe depth sensor 3577. A surfaces data establisher 3572C is instantiatedon the processor 3572, and is adapted to establish surfaces data, e.g.from one or more images received from the image sensor 3576. A notionaldata model establisher 3572D is instantiated on the processor 3572, andis adapted to establish a notional model, such as a virtual realityand/or augmented reality model.

A potential shadow identifier 3572E is instantiated on the processor3572, and is adapted to identify one or more potential shadows, forexample by color, brightness, geometry, etc. in surfaces data. A lightsource identifier 3572F is instantiated on the processor 3572, and isadapted to identify one or more light sources, for example by color,brightness, etc. in surfaces data. A light source locator 3572G isinstantiated on the processor 3572, and is adapted to locate lightsources within solids data, e.g. by projecting light sources fromsurfaces data into solids data.

An anticipated shadow determiner 3572H is instantiated on the processor3572, and is adapted to determine anticipated size, shape, position,darkness, color, etc., of shadows anticipated to exist, for examplebased on light sources and a solids model. A shadow matcher 3572I isinstantiated on the processor 3572, and is adapted to match potentialshadows against anticipated shadows to determine whether a match exists,and to identify matching potential shadows as occluding shadows.

An occluding shadow applier 3572J is instantiated on the processor 3572,and is adapted to apply occluding shadows to the notional data model. Ashadow correction establisher 3572K is instantiated on the processor3572, and is adapted to establish one or more shadow corrections for thenotional data model. A shadow correction applier 3572L is instantiatedon the processor 3572, and is adapted to apply shadow corrections to thenotional data model.

A notional content generator 3572M is instantiated on the processor3572, and is adapted to generate content reflecting the notional datamodel, suitable for output. An outputter 3572N is instantiated on theprocessor 3572, and is adapted to output notional content, for exampleto a head mounted display or other display.

The above specification, examples, and data provide a completedescription of the manufacture and use of the composition of theinvention. Since many embodiments of the invention can be made withoutdeparting from the spirit and scope of the invention, the inventionresides in the claims hereinafter appended.

We claim:
 1. A machine-implemented method, comprising in a processor:establishing a reference position; establishing a physical data modelrepresenting at least one physical entity, said physical data modelbeing spatially dynamic in time; establishing a notional data modelrepresenting at least one notional entity, said notional data modelbeing dynamic in time and non-exclusive of spatial coincidence with saidphysical data model; establishing a first sensory property from saidnotional data model; wherein said first sensory phenomenon represents aphysical environmental phenomenon; wherein said physical environmentalphenomenon comprises at least one of physical illumination, physicalshadowing, a physical volumetric effect, and an optical phenomenon; andwherein said physical volumetric effect comprises at least one of agroup consisting of at least one falling element, at least one flyingelement, and at least one suspended element; determining a secondsensory property, said second sensory property corresponding with saidfirst sensory property for another of said physical and notional datamodels; generating notional sensory content representing at least aportion of said notional data model with at least a portion of saidsecond sensory property applied thereto; and outputting to saidreference position with a perceive-through display said notional sensorycontent, registered with said physical data model.
 2. The method ofclaim 1, wherein: said first sensory property comprises at least one ofa visual property and an auditory property.
 3. The method of claim 1,wherein: said second sensory property comprises at least one of a visualproperty and an auditory property.
 4. The method of claim 1, wherein:said at least one physical entity comprises at least a portion of aviewer.
 5. The method of claim 1, wherein: said at least one notionalentity comprises at least one of a virtual reality entity and anaugmented reality entity.
 6. The method of claim 1, wherein: saidreference point corresponds with an eye of a viewer.
 7. The method ofclaim 1, wherein: said perceive-through display is a head mounteddisplay.
 8. The method of claim 1, wherein: said perceive-throughdisplay is at least one of an optical see-through display such thatoptical physical content passes therethrough and a virtual see-throughdisplay adapted to output at least a portion of said physical data modeltherefrom.
 9. The method of claim 1, wherein: said physical volumetriceffect is a function of a distance between said reference point and saidphysical data model.
 10. A machine-implemented method, comprising in aprocessor: establishing a reference position; establishing a physicaldata model representing at least one physical entity, said physical datamodel being spatially dynamic in time; establishing a notional datamodel representing at least one notional entity, said notional datamodel being dynamic in time and non-exclusive of spatial coincidencewith said physical data model; establishing a first sensory propertyfrom said notional data model; wherein said first sensory phenomenonrepresents a physical environmental phenomenon; wherein said physicalenvironmental phenomenon comprises at least one of physicalillumination, physical shadowing, a physical volumetric effect, and anoptical phenomenon; and wherein said physical volumetric effectcomprises at least one of a group consisting of ash, debris, dust, fog,gas, hail, heat distortion, insects, leaves, mist, rain, sleet, smoke,snow, spray, and steam; determining a second sensory property, saidsecond sensory property corresponding with said first sensory propertyfor another of said physical and notional data models; generatingnotional sensory content representing at least a portion of saidnotional data model with at least a portion of said second sensoryproperty applied thereto; and outputting to said reference position witha perceive-through display said notional sensory content, registeredwith said physical data model.
 11. The method of claim 10, wherein: saidphysical volumetric effect is a function of a distance between saidreference point and said physical data model.
 12. The method of claim10, wherein: said first sensory property comprises at least one of avisual property and an auditory property.
 13. The method of claim 10,wherein: said second sensory property comprises at least one of a visualproperty and an auditory property.
 14. The method of claim 10, wherein:said at least one physical entity comprises at least a portion of aviewer.
 15. The method of claim 10, wherein: said at least one notionalentity comprises at least one of a virtual reality entity and anaugmented reality entity.
 16. The method of claim 10, wherein: saidreference point corresponds with an eye of a viewer.
 17. The method ofclaim 10, wherein: said perceive-through display is a head mounteddisplay.
 18. The method of claim 10, wherein: said perceive-throughdisplay is at least one of an optical see-through display such thatoptical physical content passes therethrough and a virtual see-throughdisplay adapted to output at least a portion of said physical data modeltherefrom.
 19. A machine-implemented method, comprising in a processor:establishing a reference position; establishing a physical data modelrepresenting at least one physical entity, said physical data modelbeing spatially dynamic in time; establishing a notional data modelrepresenting at least one notional entity, said notional data modelbeing dynamic in time and non-exclusive of spatial coincidence with saidphysical data model; establishing a first sensory property from saidnotional data model; wherein said first sensory phenomenon represents aphysical environmental phenomenon; wherein said physical environmentalphenomenon comprises at least one of physical illumination, physicalshadowing, a physical volumetric effect, and an optical phenomenon; andwherein said physical environmental phenomenon comprises at least one ofa group consisting of diffraction, diffusion, glory, haloing, lensflare, and reflection; determining a second sensory property, saidsecond sensory property corresponding with said first sensory propertyfor another of said physical and notional data models; generatingnotional sensory content representing at least a portion of saidnotional data model with at least a portion of said second sensoryproperty applied thereto; and outputting to said reference position witha perceive-through display said notional sensory content, registeredwith said physical data model.
 20. The method of claim 19, wherein: saidphysical volumetric effect is a function of a distance between saidreference point and said physical data model.
 21. The method of claim19, wherein: said first sensory property comprises at least one of avisual property and an auditory property.
 22. The method of claim 19,wherein: said second sensory property comprises at least one of a visualproperty and an auditory property.
 23. The method of claim 19, wherein:said at least one physical entity comprises at least a portion of aviewer.
 24. The method of claim 19, wherein: said at least one notionalentity comprises at least one of a virtual reality entity and anaugmented reality entity.
 25. The method of claim 19, wherein: saidreference point corresponds with an eye of a viewer.
 26. The method ofclaim 19, wherein: said perceive-through display is a head mounteddisplay.
 27. The method of claim 19, wherein: said perceive-throughdisplay is at least one of an optical see-through display such thatoptical physical content passes therethrough and a virtual see-throughdisplay adapted to output at least a portion of said physical data modeltherefrom.